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LOCATING AND SEARCHING ELECTRONIC DOCUMENTS: 
A USER STUDY OE SUPPLY PUBLICATIONS 
IN THE UNITED STATES MARINE CORPS 


ABSTRACT 


This exploratory study assesses the usability of an online policy document system, 
specifically with company grade supply officers in the United States Marine Corps. 
Using common work scenarios, users were asked to search for specific policy documents. 
Their efforts to navigate through the system were recorded and analyzed to better 
understand areas in the system that posed challenges to supply officers’ efforts at policy 
retrieval and understanding. The primary purpose of this study is to provide 
recommendations, based on the test findings, that will address opportunities to enhance 
the effectiveness and efficiency of the current electronic publication system. Based on 
this analysis, the second purpose of this study was to provide recommendations to 
enhance the effectiveness, efficiency, and user satisfaction with the electronic document 
system. 

Five participants were observed as they used electronic policy and regulatory 
documents related to financial management and property control functions. The 
participants were chosen because they represent the user population and are familiar with 
the electronic document system. Each participant conducted typical search and retrieval 
tasks using think-aloud protocols. Each session was videotaped, and participants were 
interviewed afterwards. 

In-depth analysis of the data indicates that participants are generally satisfied with 
the system, but significant opportunities exist to improve its effectiveness and efficiency. 
Most notable recommendations are: increase search capability, provide consolidated 
offering of content, allow personalization and portability, improve ability to see 
context/location within documents, create additional linking mechanisms, and migrate 
away from .pdf file format. Additional recommendations address the need for iterative 
testing and triangulation of problem areas to increase data reliability. 
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I. 


INTRODUCTION 


A. PURPOSE 

A key function of a supply officer’s duties is to retrieve, interpret, and advise on 
new and existing military policy. Traditionally, such policies were housed in a paper- 
based library; however, in the past 10 years the policy libraries have been migrating to an 
electronic system, accessible via the internet. The purpose of the current study is to assess 
the usability of an online policy document system, specifically with company grade 
supply officers in the United States Marine Corps. Using common work scenarios, users 
were asked to search for specific policy documents. Their efforts to navigate through the 
system were recorded and analyzed to better understand areas in the system that posed 
challenges to supply officers’ efforts at policy retrieval and understanding. Based on this 
analysis, the second purpose of this study was to provide recommendations to enhance 
the effectiveness, efficiency, and user satisfaction with the electronic document system. 

B. BACKGROUND 

In this section, I discuss the need to study the electronic supply policy document 
system to demonstrate the effectiveness and efficiency of the existing tool. First, I 
provide an historical context by addressing the significance of supply to Marine Corps 
commands. Second, I discuss the importance of locating and retrieving policy documents 
for supply officers. Third, I provide an overview of the migration from a paper-based to 
an electronic-based policy document system. Fourth, I present the business case analysis 
that discusses the importance of usability for aiding supply officers in their use of 
electronic policy documents. Fifth, I provide an overview of usability research. Sixth, I 
present the research questions that governed this study. Finally I provide an overview of 
the project. 

Throughout this research, I shall limit discussion to Marine Corps supply officers 
and the publications and training related to their duties. Although supply functionality is 
largely executed by capable and well-trained enlisted Marines, the supply officer has a 
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direct and potentially far-reaching impact on command operations through an advisory 
role to the commander and supervisory role of supply operations. Enlisted Marines 
contribute enormously toward that goal, yet the supply officer is ultimately responsible. 
For this reason, supply officers and the policy documents they use to conduct their duties 
were selected for this study. i The participants of the study are discussed further in 
Chapter II. 

1. Significance of Supply & Supply Policy 

Supply is a crucial element in the operational readiness of any command. In fact, 
most readiness reporting includes supply readiness and maintenance readiness as primary 
measures of overall command readiness. The significance of supply is discussed in 
Marine Corps Order 4400.163, which states that “Massive manpower is not the prime 
asset of the United States; this nation’s prime asset is skilled manpower equipped with 
the proper tools.Equipping is a complex function that encompasses over $336.1 trillion 
dollars in Department of Defense (DoD) assets. Consequently, it is critical that supply 

•5 

management be effective, both for national security and economic reasons. 

Ground supply officers supervise the execution of policies and procedures that 
govern the complex function of Supply. To execute these policies, officers rely on the 
expertise and advice of enlisted personnel. Supply officers also consult with peers, 
higher authority, and others to help guide their duties. Although enlisted personnel and 
other officers assist the supply officers with policy retrieval and interpretation, proficient 
supply officers must be capable of retrieving and understanding policy documents 
themselves to conduct their functions as an advisor, trainer, and supervisor. This 
capability is necessary so that supply officers can advise and make recommendations to 
commanders regarding policy, and take corrective or preventive action when they 


^ As a Marine Corps Supply Officer for seven years, I have first-hand knowledge of the domain, 
responsibilities, and importance of the Supply Officer contribution. 

2 Marine Corps Order 4400.163, Department of Defense Supply Management Reference Book, Section 
1-2 (1985). http://www.usmc.mil/directiv.nsfA?OpenView&Count=3000 (accessed November 15, 2007). 

^ Ibid., section 1-3. 
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encounter detrimental supply situations.^ An effective policy document system, then, is 
one that will help Supply officers achieve their goal of retrieving, understanding, and 
advising others on matters of supply policy. 

2. Locating and Retrieving Policy Information 

Under the expansive policy document system, the procedural answers supply 
officers seek can be obscured by the complexity of the system itself. At a minimum, a 
user who wishes to retrieve and explore policy and regulations related to a specific task 
must: 

1. Determine relevant publications governing the task, 

2. Possess or locate the applicable publications, 

3. Search for information related to the task from within those publications, 

4. Comprehensively integrate the results, and 

5. Interpret the intent of the author 


In addition to an already complex process, the characteristics of the electronic 
document system further complicate the task. For example, the officer’s effort may be 
complicated because: a) documents originate from multiple sources and various levels of 
authority; b) policy information is parsed, segregated, or falls in overlapping topic areas; 
c) policies are in constant flux due to frequent updates, deletions, clarifications, or other 
additional alterations; and d) policies are extensively cross-referenced both within 
documents and amongst documents. Given this complexity, it is important that the 
electronic document system be as easy to use as possible. 


4 Marine Corps Order P4400.150E w/Erratum and ehange 1-2, Marine Corps Consumer Level Supply 
Policy Manual, Seetion 1003.2a (2001) (hereafter eited as MCO P4400.150E). 
http://www.usme.mil/direetiv.nsf/l?OpenView&Count=3000 (aeeessed November 15, 2007). 
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3. 


Paper-Based Policy Document System 


Historically, Marine Corps supply officers were required to have physical copies 
of all relevant publications.^ This collection of policy documents (known as a 
publication library) alleviated the burden of seeking out policy references for recurring 
tasks by providing ready reference to the policy and regulatory guidance right off-the- 
shelf. 

In some ways, the paper-based system allowed users to control aspects of 
effectiveness and efficiency. By physically manipulating their personal copies, users 
were able to organize them in a manner they deemed useful and efficient. Users could 
also bookmark, highlight, and tab relevant or repetitive areas of interest for future use. 
Furthermore, the paper-based system provided users the ability to search for information 
in familiar ways that are traditionally employed with paper documents and books (e.g., 
indexes, table of contents, scanning). A major drawback to this system, however, was the 
need to constantly maintain the publication library. Maintaining the system took a 
significant amount of time, and publications were often missing change notices, updates, 
or complete documents. 

4. Electronic Policy Document System 

The advent of the internet brought forth a new and more useful medium through 
which the Department of Defense (DoD) could distribute its policies. In a policy 
statement issued in 1998, the DoD Web Site Administration Policy and Procedures 
proclaimed, “It is the policy of the DoD that...using the World Wide Web is strongly 
encouraged in that it provides the DoD with a powerful tool to convey information 
quickly and efficiently on a broad range of topics relating to its activities, objectives, 
policies and programs.”® 


5 Marine Corps Order P4400.150E 

^Department of Defense, Web Site Administration Policies and Procedures, Part I. 4.1 (1998). 
http://www.defenselink.mil/webmasters/poliev/dod web poliev 12071998 with amendments and corree 

tions.html (aeeessed December 6, 2007). 


4 





The Marine Corps embraced this move toward paperless policy documents in 
1997. It was then that the Corps began revising its printing and publications regulations 
to take advantage of new technologies, and to comply with the Paperwork Reduction Act 
and the Assistant Commandant of the Marine Corps (ACMC’s) Paperwork Reduction 
Initiative.^ In 1998 the regulation was updated to authorize electronic publication 
libraries rather than printed libraries. And, in 2001, a further update announced, “The 
Marine Corps homepage at http://www.usmc.mil is the official Marine Corps web page to 
access Marine Corps directives online.”^ These announcements authorized electronic 
access alone as sufficient to meet the requirement for on-hand publications libraries as 
long the documents were kept up-to-date. The policies further stipulated that the internet 
could be used as a source as long as the digitized fdes were downloaded and kept 
locally.9 

Many units have since abandoned maintenance of a paper-based library in favor 
of an electronic library. This migration is in keeping with the DoD and Marine Corps 
trend of converting (or originating) the majority of publications and documents to digital 
format and making them available for use electronically. Steve Sherman, Director of the 
Document Automation and Production Service (DAPS),!*’ acknowledges this trend on the 
DAPS website, stating “online document services are now the largest part of our 
business. They have increased by 114 percent in the past seven years.”” 

5. Business Case for Usability 

Regardless of the medium, the goal of policy is to ensure compliance with 
procedure, and supply officers use policy documents as a primary resource to achieve that 

^ Marine Corps Order P5600.31G w/ Change 1-3, Printing and Publications Regulations, Background 
(2001). http://www.usmc.i'nil/directiv.nsf/gam?OpenView&Count=3000 (accessed October 5, 2007). 

^ Ibid., part 3113. 

Ibid., part 3206.1. 

!*! DAPS is a division of the Defense Logistics Agency charged with providing effective and efficient 
document services support to the DoD components. Department of Defense Instruction 5330.03, Document 
Automation and Production Service (February 2006). 

!! Document Automated Printing Service website, 
http://www.daps.dIa.miFDAPS MORE THAN A PRINTER.asp (accessed September 26, 2007). 
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goal. As previously noted, supply officers have traditionally used a paper-based 
document system that afforded them the opportunity to personalize documents, to have 
them readily available, and to search and navigate them in familiar ways. Today, supply 
officers have an electronic capability to achieve their goals. However, using an 
electronic system may be challenging if the system does not properly address the 
differing aspects of electronic and paper-based documents. 

Significant research exists regarding the differences in electronic and paper-based 
documents. In a study of hypertext documents, for example, researchers showed that if 
electronic documents merely imitate paper, the paper documents will be preferred. The 
research also showed that readers who feel disoriented, worry they are overlooking 
crucial information, or cannot use the search mechanism will either abandon the 
electronic document for paper, or abandon the task altogether. 12 Other research argues 
that electronic documents “must be at least as easy to use as a printed book.”i3 
Therefore, merely scanning paper documents onto the web may not fully maximize the 
intent of the DoD policy of conveying information quickly and efficiently, and users may 
opt instead for the old paper-based system. 

A more effective, efficient, and satisfying electronic document system should 
incorporate well-designed documents and institute advantages of online documents 
available via the internet. For example, one advantage is keyword searching, which 
enables an officer to find a specific area of interest in a large document, or in a collection 
of documents, within seconds. Another advantage of online documents is the ability to 
hyperlink to related areas within or between documents, which may further enhance the 
officer’s ability to access cross-references more efficiently. These features and others of 


Florence M. Pillion and Craig D. B. Boyle, “Important issues in hypertext documentation 
usability,” In Proceedings of the 9th Annual international Conference on Systems Documentation 
(Chicago, Illinois, 1991). SIGDOC '91. ACM, New York, NY, 59-66. DOH 
http://doi.acm.org/10.1145/122778.122788. 

12 Laurie Kantner, Roberta Shroyer, and Stephanie Rosenbaum, "Structured heuristic evaluation of 
online documentation," In Professional Communication Conference, (2002). IPCC 2002. Proceedings. 
IEEE International, 331-342. 

http://ieeexplore.ieee.org/iel5/8118/22469/01049115.pdf?isnumber=22469n=STD&amumber=1049115&a 

mumber=1049115&arSt=-r331&ared=-r342&arAuthor=Kantner%2C-rL.%3B-rShrover%2C-rR.%3B-rRose 

nbaum%2C-rS. 
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an electronic document system can reduce the burden of policy retrieval, potentially 
resulting in a higher level of policy compliance. 

Additionally, these features need to be well-designed and incorporated in an 
overall system that aids the supply officer in finding policy information efficiently. For 
example, an ineffective search engine that provides unexpected results may be far less 
effective than a paper-based system: in short, the features must support the specific needs 
of the user. If the Marine Corps intends for the supply community to reap benefits from, 
and fully embrace, an electronic supply policy document system, the system must be one 
that works effectively and efficiently. An exploratory usability study is one method to 
assess the effectiveness of a system and to provide a basis for recommended changes. 

6. Overview of Usability Research 

The International Organization for Standardization (ISO 924I-II), defines 
usability as: 

The extent to which a product can be used by specified users to achieve 

specified goals with effectiveness, efficiency and satisfaction in a 

specified context of use.i^ 

There are three critical issues embedded in this definition: 

1. Usability tests are focused on a product 

2. Usability is defined and measured along three axes 

3. Usability tests have specific users, tasks, and contexts 


First, the goal of usability tests is to improve the product, not explain or predict 
phenomena through inference or extrapolation of data. Usability practitioners Dumas and 
Redish define this goal as follows: 


14 


International Organization for Standardization, Guidance on Usability, ISO 9241-11 (1998). 
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The primary goal of a usability test is to improve the usability of the 
produet that is being tested...This characteristic distinguishes a usability 
test from a research study, in which the goal is to investigate the existence 
of some phenomenon. 

Second, usability is defined and measured along three specific criteria: 
effectiveness, efficiency, and satisfaction. These elements are defined as follows: 


• Effectiveness : The user’s ability to achieve specific goals in the 
environment 

• Efficiency : The resources used (time, money, and mental effort) when 
performing a system-supported task 

• Satisfaction : The user’s comfort level and acceptance of the system 

overall 15 


These three factors can be measured in a variety of ways that may include, for 
example, tracking of errors encountered [effectiveness], number of mouse clicks to a 
defined destination [efficiency], time spent on a task [efficiency], confusion/frustration 
observed [satisfaction], and many others.i^ (The specific measures that I examined for 
this study, and their accompanying metrics, are discussed in more detail in Methods, 
Chapter II). 

Einally, a usability test is distinguished by its focus on specific users doing 
specific tasks in specific contexts of use. When designing usability tests, the usability 
practitioner must define who the users are, what common tasks are to be investigated, 
under what conditions the test will occur, and how usability will be measured. Eor 
example, in this study, I am interested in how supply officers, in their daily work, locate 

15 Joseph S. Dumas and Janice C. Redish, A Practical Guide to Usability Testing (Norwood, NJ: 

Ablex Publishing, 1993), 22. 

16 Ibid. 

l'^ Ibid, 185. 
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electronic policy documents. I therefore selected active supply officers, situated the test 
at the participants’ work space, and asked them to do common representative tasks (A 
more detailed description of criteria used to select appropriate participants, tasks, and key 
interactions is discussed in Methods, Chapter II). 

There is some debate in the literature about how many users are necessary to 
derive valid results. However, studies have shown that five users can discover 80% of 
usability problems. Usability expert Jakob Nielsen pioneered a 1989 study that 
established a precedent for the required number of users for what he termed a “discount” 
usability study. He argued that “elaborate usability tests are a waste of resources. The 
best results come from testing no more than five users and running as many small tests as 
you can afford.”i9 He argues that after six users, there is a clearly diminished return for 
number of problems detected. As Figure 1 (below) indicates, there is a negligible gain of 
problem identification between 6 and 9 users, and almost no gain between 9 and 15 users. 

More recently, some research has challenged the assertion that five participants 
are enough for a sufficient study. In 2001, Jared Spool’s report contradicted the model 
after conducting a series of website tests which only indicated identification of 35% of 
problems after five users, and some serious usability problems were not identified until 
after the thirteenth or fifteenth user.^o 

Despite these challenges, however, the purpose of usability testing must be taken 
into consideration—usability is about improving the design of a product. Even Spool 
acknowledges that any number of users can help uncover issues: “For whatever value of 
‘N’ [number of users] they [design teams] choose, ‘N’ users will always be better than 


Jakob Nielsen, “Usability engineering at a discount.” In Proceedings of the Third international 
Conference on Human-Computer interaction on Designing and Using Human-Computer interfaces and 
Knowledge Based Systems (2nd Ed.) (Boston, Massachusettss). G. Salvendy and M. J. Smith, Eds. Elsevier 
Science, New York, NY, 394-401. 

Jakob Nielsen’s usability Web site, http://www.useit.com/alertbox/20Q0Q319.html (accessed 
October 10, 2007). 

20 Jared Spool and Will Schroeder, “Testing web sites: five users is nowhere near enough.” In CHI '01 
Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington, March 31 - April 05, 
2001). CHI '01. ACM, New York, NY, 285-286. DOI= http://doi.acm.org/10.1145/634067.634236 
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zero.”21 Other types of studies may net greater findings, but can be very costly. And, as 
Neilsen argues, a considerable number of problems can be detected even with only a few 
participants. No study will be “perfect,” he noted, nor will it “discover everything that’s 
possible to know about the design, but we accept this trade-off in return for having more 

99 

iterations in the design process.” 



Number of Test Users 

Figure 1. Usability Findings by Number of Users23 

In this usability study, there is no expectation that all the usability problems will 
be uncovered. However, as Neilsen points out, we can still expect to gain valuable 
insights from which we can derive recommendations. 


21 Nigel Bevan and others, “The ‘magie number 5’: is it enough for web testing?” In CHI '03 
Extended Abstracts on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA, April 05 - 10, 
2003). CHI '03. ACM, New York, NY, 698-699. DOI= http://doi.aem.org/10.1145/765891.765936 

22 Ibid. 

23 See note 19. 
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7. 


Research Questions 


For this study, I identified three speeifie researeh questions related to how well 
supply offieers eould retrieve online poliey information: 

1. Can users easily and effieiently loeate finaneial management and property 
eontrol regulations? 

2. Can users easily and effieiently loeate all neeessary information to answer 
work-related questions? 

3. How do users look for information? 

8. Preview of Project 

Following this introduetory ehapter, the rest of the projeet is divided into three 
additional ehapters. Chapter II presents the methods I used in preparing for and 
eondueting the usability test. Chapter III presents the findings from the study, ineluding 
direet quotes from partieipants as well as observations of user behavior. Chapter IV 
provides reeommendations for ehanges to the system based on the findings derived from 
testing and user feedbaek. 


II 
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II. METHODS AND PROTOCOLS 


This chapter details the proeedures used to conduct the usability test. In the 
Methods section, I: 

• Provide an overview of the test 

• Describe the test participants 

• Describe users’ key tasks 

• Present the underlying researeh questions 

• Present the test scenarios 

• Present the metrics and protocols used for analysis 

In the Protoeols seetion, I deseribe speeific approaehes used in eollecting and 
analyzing the survey and verbal data. 

A. METHODS 

1. Overview 

The usability test involved five partieipants who would regularly use the system 
to retrieve policy documents online in the daily conduet of their work. For the purposes 
of this study, I defined the online system as “the eollection of official policy source 
doeuments and any means used to electronieally retrieve those documents.” This 
definition allowed me to explore the user’s natural approach to locating and searching 
electronic policy documents. 

I chose partieipants based on preliminary research of users of the documents; I 
reviewed demographic data for the Marine Corps and conducted a preliminary survey. 
The preliminary survey focused on identifying eommon information retrieval tasks and 
underlying user goals. 
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During the test, participants were each given two scenarios that required them to 
navigate and search online policy documents. While executing specific tasks, 
participants were asked to “think aloud” while the researcher directly observed and 
video-taped the sessions. Upon completion of the specified tasks, the participants 
completed a 15-minute interview followed by a short, 10-question, satisfaction interview 
questionnaire known as the System Usability Scale (SUS) (See Appendix A). The 
interview and SUS explored participants’ experience with and evaluation of the system. 

2. Preliminary User Survey 

Prior to designing the usability test, I wanted to understand the key drivers for 
information retrieval. To help me identify the appropriate participants and tasks, I 
created a short survey instrument (see Appendix B) that identified common policy areas 
and specific tasks that Marine Corps supply officers would likely encounter. For 
example, I selected key areas such as property control or requisition management. 
Within each area, I also provided a selection of tasks that would be appropriate to that 
area, such as sub-custody receipting (CMRs) or inventory would be for the property 
control area. Drawing from the Total Force Data Warehouse (TFDW) database^^, I 
identified 258 potential participants and contacted them via email. Of the 258 number 
contacted, 109 responded. The survey asked respondents to select the policy area they 
most frequently consulted and the primary tasks within those policy areas that they most 
frequently conducted. Additionally, I asked them to identify the three most common 
reasons for using electronic publications. Finally, I used the survey as a recruiting tool; 
the final question asked if they were willing to be contacted to be a participant in the 
study. Results are summarized in Table 1, below: 
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TFDW accessed via Marine Online website, http://mol.usmc.mil (accessed September 1, 2007). 

14 




Table 1. Summary of results from preliminary user survey 


Response 

Three Most Common Reasons for Consulting Policy Documents 

59.8% 

Advise Commander 

57.8% 

Prove/Justify Procedures to an Interested Party 

53.9% 

Verify Personal Action/Procedure Complies with Reference 

Response 

% Time Spent Seeking Policy Information in Electronic Documents 

41.2% 

10% - 20% 

22.7% 

Less than 10% 

20.6% 

20% - 30% 

Response 

Area and (Task within that area) Requiring the Most Time 

30.4% (64.5%) 

Financial Management (Government Commercial Purchase Card) 

29.4% (44.8%) 

Property Control (CMR) 


I used the results of this preliminary survey to select test participants and to 
choose test scenarios and tasks. 

3. Participants 

In usability tests, it is important to be sure that our test participants align with our 
business and practical goals. In this current study of electronic supply documents, 
possible user populations included supply personnel, government contractors, legislative 
bodies, commanders, and many others. However, for this usability test, I focused 
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specifically on intermediate-level officer leadership (company grade officers) within the 
Marine Corps. I selected this representative user group for four primary reasons: 

1. They have a large impact on the organization 

2. Their characteristics are typical of the targeted user base 

3. They have varying degrees of familiarity with the interface 

4. They were readily and locally available 


Additionally, based on the rationale provided by Neilsen’s standard-setting 
research showing that nearly 80% of usability problems can be found by testing only five 
users, 251 targeted five users for this test. 

a) Supply Officers’ Impact on the Organization 

When selecting test participants, I wanted to ensure that the participants 
were key users of the system. When looking at the various roles associated with the use 
of supply policy documents, supply officers stood out as key business users. As advisors 
to the Commander, trainers of junior officers, and supervisors of enlisted men, supply 
officers have a large impact on their command. First, supply officers are directly charged 
with advising and recommending corrective procedures based on policies, laws, and other 
regulatory guidance contained in supply policy documents. Second, the Marine Corps 
mandates that supply officers establish an “ongoing and effective training program” on 
policies and procedures. Even though actual training is often conducted by subordinate 
leaders, the overall responsibility for creating the training is explicitly that of the supply 
officer. Finally, given that there is approximately one officer for every ten enlisted 


25 See note 18 above. 

26 See note 4 above. 

27 MCOP4400.150E, Seetion 1011. 
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Marines, the supply officer has significant supervisory and oversight responsibilities 
related to policy understanding and compliance. 

b) Supply Officers are Typical of Targeted User Base 

In usability testing, it is important to select your test participants based on 
the alignment of their personal characteristics with the key user population. Key 
characteristics to look at include personal demographics and skill levels. As part of our 
recruiting process, we verbally verified that participants met the desired characteristics 
prior to scheduling them. In addition, we verified their characteristics prior to 
administering the test. 

For this study, I selected participants that would closely match the 
demographics of supply officers. Marine Corps officers are largely homogenous in terms 
of education, rank, and age. In addition, occupational assignments within the Marine 
Corps are not tied to skills—all supply officers receive formal supply training at the 
Ground Supply Officer Course (GSOC). Therefore, we selected participants based on the 
following Marine Corps demographic trends: 

• 90% hold a BA degree or higher 29 

• 55% of total supply officers are captains or first lieutenants 

• 47% are between the ages of 26 and 3530 

• Captains predominantly have 4-10 years military experience and first 

lieutenants have 2-4 years experience^! 


28 Headquarters, United States Marine Corps, 2007 Marine Corps Almanac, Headquarters Marine 
Corps website (2007). http://hqinet001.hqme.usme.mil/p&r/coneepts/2007/CHPT5.htm (aeeessed 
September 18, 2007). 

29 Marine Corps Community Serviees, Marine Corps Community Services Demographics Update 
(June 2007). http://www.usme-mecs.org/aboutmecs/downloads/pom/Demographies%20Update.pdf 
(accessed October 12, 2007). The data for the report is extracted from the Marine Corps Total Force Data 
Warehouse. 

2!! See note 28, above. 

3! Department of Defense Instruction 1320.13, Commissioned Officer Promotion Reports (COPRs) 
and Procedures, (Junel996). http://wwwO.dtic.mil/whs/directives/corres/html/132013.htm (accessed 
October 12, 2007). 
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c) Supply Officers Have Varying Familiarity with the Interface 


Within my five participants, I recruited officers who were both more 
familiar and less familiar with retrieving policy information online (see Table 2, below). 
These participants represented the larger user population whose responses in the 
preliminary survey indicated similar variance in familiarity. Specifically, 41.2% of 
respondents indicated that they spent between 10% and 20% of their time looking for 
information in electronic publications, and another 22.7% indicated that they spend less 
than 10% of their time doing so. Therefore, selecting participants that had different 
levels of familiarity with the system gave a better approximation of the actual user group 
as a whole. 

More junior officers (second lieutenants) were excluded from the user 
population since their lower job experience level may have strongly and negatively 
impacted their performance. However, these more novice users may be an important 
secondary audience to consider in future studies. 

d) Participant Availability 

Finally, while I tried to recruit representative participants, the short time 
frame for this study, and the resources involved in soliciting from a larger participant 
pool, meant that I had to limit my participant base to California. All participants were 
selected from Marine Corps Base, Camp Pendleton due to its proximity and ready access 
to users. However, to ensure that our participants were appropriate, we included a brief 
pre-test questionnaire (see Appendix C) to verify that their characteristics were consistent 
with the target user profile derived from Marine Corps demographics: Captains and First 
Lieutenants between the ages of 26 and 35 with no less than two years of service and a 
minimum four-year college degree. Table 2 summarizes the actual characteristics of our 
five test participants. 
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Table 2. Pre-Test Survey Results 


Stats from Pre-Test Questionnaire 


Aqe Group 

Age 26-35 

All Users 

100% 





Internet familiarity 

Very familiar 

User 1,4 

40% 


Familiar 

User 2,3,5 

60% 





Time spent on pubs 

0-10% 

User 1,4,5 

60% 


10%-20% 

User 3 

20% 


30%-50% 

User 2 

20% 





How lonq in supply? 

yrs 2-4 

User 4,5 

40% 


yrs 4-10 

User 1,2,3 

60% 





Education 

BA/S degree 

User2,3,4,5 

80% 


graduate degree 

User 1 

20% 


4. Selecting Appropriate Tasks 

Tasks conducted during a usability test must represent tasks that the defined users 
would realistically be expected to perform; this requirement necessitates an 
understanding of users’ jobs and tasks.^2 The results of the preliminary survey (see Table 
1, page 15) provided valuable insights and helped me to better understand user tasks. 

Driven by these survey responses and my own experience with practical situations 
in those areas, I created two scenarios based on the most common tasks selected by 
respondents^^ (a synopsis of the scenarios is provided later in this chapter). 

5. Targeting Components 

A usability test, depending on its goals, targets certain components of the overall 
system. Typical components include help systems, navigation systems, page layouts, and 
others. However, testing every component of a system can be cost-prohibitive and time- 

32 Dumas and Redish, A Practical Guide to Usability Testing, 23. 

33 Test results indicated that users found the selected tasks to be relevant and realistic of those they 
would conduct on the job (See chapter III). 
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consuming. In this study, I was interested in understanding how users “naturally” use the 
system to locate and retrieve information; in other words, users needed to be free to 
navigate and search as they would normally. Therefore, for this test, I seleeted 
eomponents which are key to locating information—information navigation and search. 

a) Information Navigation 

The poliey doeuments, as previously indicated, come from multiple 
sourees, in a variety of policy document types, and have frequent changes, supplements, 
and eaneellations. Additionally, many Web sources make internal referenees and cross- 
references to other policy documents whieh may or may not be loeated under the same 
source. This system eomplexity may impede the users’ ability to find the doeuments they 

need, and to collate the information in an effective and effieient manner. To understand 
navigation diffieulties better, I focused on instanees where a user eannot accomplish the 
following; 

• Identify the correct location 

• Return to a previous location 

• Find information believed to exist somewhere, or 

• Remember key points learned^^ 

If a navigational aid is effective, it should assist the user in finding 
information effectively and efficiently. 

b) Search 

A primary benefit of electronie resources is that systems can improve how 
users can search for information, which can positively impact effectiveness and 
effieiency. Keyword searehing, hyperlinking, and other methods have brought new 
aeeess capabilities when eompared with paper-based systems. Seareh meehanisms are 

Deborah M. Edwards and Lynda Hardman, “Lost in hyperspace: cognitive mapping and navigation 
in a hypertext environment” In Hypertext: theory into Practice, R. McAleese, Ed. Intellect Books, (Exeter, 
UK, 1999), 90-105. 
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particularly valuable with regard to policy documents where the user’s primary purpose is 
to retrieve specific policy information rather than simply browsing or reading for 
enjoyment. In this study, I targeted the eomponent of seareh mechanisms because of its 
partieular importance to my users’ key tasks. 

B. RESEARCH QUESTIONS AND USABILITY MEASUREMENTS 

Seenario tests should be based on realistie seenarios that drive to speeifie researeh 
questions. For this study, I identified three speeifie researeh questions that foeused on 
basie information retrieval (question 1) and on the larger issues of goal (question 2) and 
proeess (question 3): 

1. Can users easily and effieiently loeate finaneial management and property 
eontrol regulations? 

2. Can users easily and efficiently loeate all neeessary information to answer 
work-related questions? 

3. How do users look for information? 

I. Usability Measurements 

Each of the above research questions were broken down into key usability 
elements around the issues of navigation and seareh. I have also identified the primary 
modes of answering these researeh questions (indieated parenthetieally). 

a) Issue 1 — Effectiveness 

Issue 1.1 Do the navigation and seareh tools effeetively aid users in loeating 
a Marine Corps poliey doeument that will help them aehieve their 
goals? (task one failure rates - property eontrol seenario) 
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Issue 1.2 


Do the navigation and seareh tools effectively aid users in locating 
policy documents that originate from other sources? (task three 
failure rates - financial management scenario) 

Issue 1.3 Do search tools and navigational design within documents provide 
an effective means for users to locate critical information related to 
their goals? (task two and task four failure rates, verbal data) 

Issue 1.4 Degree to which users believe the system supports their work 
(subjective scale question) 

b) Issue 2 — Efficiency 

Do navigation design and search tools assist users by enabling easy 
and efficient location of policy documents? (number of methods 
used/paths followed, failed keyword searches, verbal data) 

Do navigation design and search tools assist users by enabling easy 
and efficient searching for information within documents? (number 
of methods used/paths followed, failed keyword searches, verbal 
data) 

c) Issue 3 — Satisfaction 

Issue 3.1 Overall user confidence in the system (subjective scale questions. 
System Usability Score (SUS), verbal data) 

2. Summary 

The following summary table provides a graphic illustration of how the scenarios 
relate to the areas of concern (issues) and the components for which I desire to gain 
insight through the testing. Additionally, the methods of observation for each scenario 
are also provided; 


Issue 2.1 


Issue 2.2 
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Table 3. Project Overview 


Task 

Scenario 

Issue 

Components 

Observation 

1 

• Electronically locate 
supply policy 
documents to guide 
you in accounting for 
SL-3 items 

• Easily Locate Marine 
Corps documents 

• Currencv/source 
reliabilitv 

• Satisfaction 

• Navigation Aids 

• Search tools 

• Task failure rates 

• Survey response 

• Verbal 

• SUS 

2 

• Use electronic supply 
policy documents to 
determine authority to 
establish quantity, and 
procedures for 
accounting for UURI 
SL-3 items 

• Locate critical 
information 

• Easily search critical 
information 

• Satisfaction 

• Navigation Aids 

• Search tools 

• Task failure rates 

• Methods used 

• Failed Searches 

• Verbal 

3 

• Electronically locate 
governing GCPC 
policy documents 

• Easily locate 
documents from 
another source 

• Currencv/source 
reliability 

• Satisfaction 

• Navigation Aids 

• Search tools 

• Task failure rates 

• Survey response 

• Verbal 

• SUS 

4 

• Use electronic GCPC 
documents to 
determine authority to 
purchase 

• Locate critical 
information 

• Easily search critical 
information 

• Satisfaction 

• Navigation Aids 

• Search tools 

• Task failure rates 

• Methods used 

• Failed Searches 

• Verbal 


Each of my research questions, along with the targeted tasks and components, 
drove the design of the test scenarios. My goal was to create test scenarios and tasks that 
would drive users to suspected usability problem areas and provide the basis for 
measurements of usability.35 


35 Dumas and Redish, A Practical Guide to Usability Testing^llS. 
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3. 


Scenarios and Tasks 


The following section provides a synopsis of the scenarios and tasks given to 
participants. The researcher created the tasks in accordance with guidance from A 
Practical Guide to Usability Testing 36 that cites four criteria for selecting tasks: 

• Tasks that probe potential usability problems 

• Tasks suggested from your concerns and experience 

• Tasks derived from other criteria 

• Tasks that users will do with the product 

The preliminary survey results indicated that the two tasks most commonly 
conducted were in the areas of property control and financial management. So, 
participants were given two scenarios, one in each of the two areas. After the test, 
participants confirmed that the scenarios were realistic. As one participant put it, “these 
are great scenarios...because I’ve been there.” Another participant noted, “It’s funny. 
I’ve actually seen stuff like this before,” and still another echoed that, “Fve had personal 
experience with something similar.” The complete details of the scenarios and tasks can 
be found in the test plan in Appendix D. 

a) Property Control Scenario 

The property control scenario required the participant to investigate proper 
accountability procedures regarding “Using Unit Responsibility Item” (UURI). The 
specific instructions required the participant to advise the commander as to whether or 
not specific items are to be recorded on supply property records and, if so, who would be 
authorized to establish an allowance for such an item. 

Task One. The first task under this scenario required participants to locate 
policy documents they thought were applicable. Participants may or may not know the 
title or numerical designator of the policy source when executing the task. The task 


36 Ibid, 160. 
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required seareh and navigation to a speeifie internal Marine Corps supply proeedure. 
Partieipants were given a five minute time limit to eomplete this task. The task 
speeifieally targets the user’s ability to loeate a partieular Marine Corps poliey doeument 
that will help them aehieve their goals (issues 1.1 and 2.1). 

Task Two. The seeond task for this seenario targets a more eomplex 
issue—^whether or not users ean use navigation and seareh aids within a doeument to 
effeetively and effieiently loeate and eollate information into a eohesive response to a 
poliey-related question (issues 1.3, 1.4, 2.2). A time limit of ten minutes applied to this 
task. 


b) Financial Management Scenario 

The finaneial management seenario required the partieipant to determine 
the legality of a proposed purehase of un-prieed maintenanee serviees for a General 
Serviees Administration (GSA) vehiele with the Government Commereial Purehase Card 
(GCPC), given an estimated eost of $2600. 

Task Three. Task three falls under the finaneial management seenario and 
essentially mirrors task one in the preeeding seenario; the partieipant must loeate the 
applieable eleetronie poliey doeuments within a five minute time limit. The 
differentiating faetor is that the GCPC program is not governed direetly by the Marine 
Corps and thus tests how users might look for poliey doeuments when the souree is less 
obvious. Therefore, this task targets the user’s ability to loeate a poliey doeument that 
originates from a souree other than the Marine Corps and that will help aehieve the user’s 
goals (issue 1.2 and 2.1). 

Task Four. Task four targets the user’s ability to easily and effieiently 

seareh for information within doeuments to answer a work-related question. In this 

seenario, partieipants were asked to identify three items: whether the purehase exeeeds 

the purehase amount threshold, if the poliey prohibits a purehase of this type (un-prieed 

serviee), and if any authority prohibits the purehase of this partieular item (GSA vehiele). 

A time limit of ten minutes applied and the resulting data were used to address researeh 

issues relating to the effeetiveness of seareh tools and navigational design within 
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documents (issue 1.3), the effieieney of those navigation and search tools (issue 2.2), and 
overall user eonfidenee in the system (issue 3.1). 

4. Metrics 

In this seetion, 1 define metries that 1 used to evaluate performanee ineluding 
failure rates, verbal data. System Usability Scale (SUS), number of paths followed, and 
failed keyword searehes. 

a) Failure Rates (Task 1 - Property Control Scenario) 

Task suecess was defined as a partieipanf s retrieval of a poliey doeument 
whieh eould provide the answers required in task two within a time limit of five minutes. 
Time began after partieipants had read the seenario and began to use the eomputer. The 
primary policy document whieh provides these answers is Marine Corps Order 
P4400.150E, the Marine Corps Consumer Level Supply Policy Manual, located on the 
Marine Corps homepage at www.usme.mil . There are multiple ways in whieh the 
document could have been aeeessed. One possible navigational approach was to select 
‘Publications’ from the main screen menu, and then select ‘Orders and Directives.’ This 
would lead the user to another sereen whieh lists methods of browsing doeuments. From 
there, seleeting ‘MCO “P” Direetives’ would provide the entire listing from whieh the 
user eould seleet this document. 

b) Failure Rates (Task 2 - Property Control Scenario) 

Partieipants who were unsueeessful in loeating a poliey doeument during 
task one were given MCO 4400.150E as a starting point for task two. Task sueeess was 
determined by the partieipanf s eorreet response to the two requirements of the task 
within a ten minute time limit. A eorreet response to the first question required 
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identification of the commanding officer as the proper authority to establish an allowance 
for a type II UURI SL-3 item. This information was located in paragraph 2011c, which 
states: 

...where “AR” (as required) is the stated quantity, the commander must 
establish, in writing, the authorized quantity to be held by the command.^^ 

The correct response for item two was located in the next paragraph 
201 lc(l), which stated: 

UURI’s that are identified as having a type 1 or 2 TAMCN [Table of 
Authorized Material Control Number] should be validated against the 
unit’s EAF [Equipment Allowance Eile] and accounted for on the unit 
property records.38 

c) Failure Rates (Task 3 - Financial Management Scenario) 

Task success was defined as a participant’s retrieval of a policy document 
which originated from an official government source, and which could provide the 
answers required in task four, within a time limit of five minutes. Time began after 
participants had read the scenario and began to use the computer. The primary policy 
document which governs the Government-Wide Commercial Credit Card Program 
(GCPC) within the United States Marine corps is NAVSUPINST 4200.99, issued by the 
Department of the Navy. 39 A variety of other documents also provide policy guidance 
for the program, and originate from official government sources, including Standard 
Operating Procedures (SOPs), Guidebooks, Internal Operating Procedures (lOPs) from 
subordinate commands and many others. 

At the time of the test, two primary options existed to retrieve an 
acceptable policy document. One option was the Marine Corps’ Contract Management 

37 MCOP4400.150E, Section 2011c. 

38 Ibid., section 201 lc(l). 

39 Naval Supply Instruction (NAVSUPINST) 4200.99, Department of the Navy (DON) Policies and 
Procedures for the Operation and Management of the Government-Wide Commercial Purchase Card 
Program (GCPC), (October 2006) (hereafter cited as NAVSUPINST 4200.99). 
http://hqinet001.hqmc.usmc.mil/I%26LW2/CMPG/ (accessed October 18, 2007). 


27 




Process Guide (CMPG) (NAVSUPINST 4200.99), located at 
http://hqinet001.hqme.usme.mil/i&L/v2/CMPG/index.htm . Additionally, the Marine 
Corps Base, Camp Pendleton, GCPC program is a source for this information.^o 

d) Failure Rates (Task 4 - Financial Management Scenario) 

Participants who were unable to locate a document in task three were 
given an eleetronic pdf file of NAVSUPINST 4200.99 retrieved from the Marine Corps 
CMPG site identified in paragraph (b) above, as a starting point for task two. Task 
success was determined by the participant’s determination that the pure base was 
unauthorized by explieitly referencing one of the three faetors whieh would have 
prohibited the purehase and doing so within a time limit of ten minutes. The three faetors 
were: 

1. Dollar amount exceeds purchase limit threshold,'^! 

2. An un-prieed serviee without an established eeiling price is 
unauthorized,42 or 

3. The specific exclusion of purchasing services for GSA-leased vehicles.^3 


Additionally, the information must have been derived from an offieial 
government source as defined in paragraph (b) above. There were three aeeeptable 
solutions: 


Regional Contracting Office - Southwest (Marine Corps Base, Camp Pendleton, CA). 
http://www.rcosw.com/apporders.htm (accessed October 10, 2007). 

41 NAVSUPINST 4200.99, section l.d.b.l. 

42 Ibid., Section 5.19. 

43 Regional Contracting Office Southwest, Government Commercial Purchase Card (GCPC) 
Program Internal Operating Procedures,{ Marine Corps Base Camp Pendleton, California) (September 
2003). http://www.rcosw.eom/RCO-SW%20%20IOP.pdf (accessed October 18, 2007). 
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1. Using NAVSUPINST 4200.99 or other official guide books which 
prohibit un-priced services and sets limits to the purchase price. 

2. Using Purchase Card Policy Notice (PCPN) 7 to identify that the purchase 
exceeded the recently increased purchase limit established. 

3. Using the Regional Contracting Office - Southwest (RCO-SW) website 
was accessible at www.rcosw.com . to see that the ‘repair or alteration of a 
GSA-leased vehicle’ is prohibited for card holders under their purview. 

e) Verbal Data 

Verbal data are qualitative measurements collected in accordance with the 
protocols identified later in this section. Verbal data, to include both think-aloud 
commentary and interview responses (discussed at length in the Protocol section, below), 
were assessed and attributed to specific problem areas where applicable. 

f) System Usability Scale (SUS) 

The SUS survey tool was developed in 1986 by usability practitioner John 
Brooke.^^ The survey consists of a series of ten questions, each using a five-point Likert 
scale to rate varying aspects of usability. Scoring is combined to provide a single 
measure of overall usability. To determine the SUS score, I used the standard scoring 
process described by the designer of the SUS, resulting in a usability rating on a 0% - 
100% scale. The protocol and its specific scoring mechanism are discussed fully in the 
Protocol section below. 

g) Number of Paths Followed 

Two types of paths were tracked—searching and navigating. Searching 
different keywords within a single search technique was not considered a different 
approach. However, applying a different search filter or different search engine was 
considered a different search approach. Additionally, if a search approach was used and 


29 



abandoned for another seareh approaeh, returning to the original approaeh would qualify 
as an additional instance when calculating the number of search paths used. 

The other approach possible was a navigational trail. An instance of a 
navigational trail was recorded for each “wrong” path of navigation. “Wrongness” was 
determined by the participant’s election to ‘go back’ and begin a different navigational 
trial, or by simply abandoning the location for another method. 

In total, the number of paths used consists of the combined number of 
instances for both approaches. 

h) Failed Keyword Searches 

A failed keyword search was recorded each time a participant’s keyword 
did not bring him to a location which could provide success as identified in each of the 
task areas. 

In addition to the above metrics, I employed several different protocols 
which are discussed in the next section. 

C. PROTOCOLS 

Protocols used, in addition to the above metrics, include surveys, semi-structured 
interviewing, and “think-aloud” verbalizations as data sources. 

1. System Usability Scale (SUS) and Other Subjective Scale Questions 

In 1986, usability practitioner John Brooke developed the System Usability Scale 
(SUS), as a reliable, low cost scale that can be used for global assessments of systems 
usability.45 Each question on the SUS uses a five point Likert scale, soliciting the degree 
to which a participant ‘strongly agrees’ or ‘strongly disagrees’ with statements regarding 


44 John Brook, “SUS: A ‘Quick and Dirty’ Usability Scale.” In Usability Evaluation in Industry, ed. 
Patrick W. Jordan (London: Taylor and Francis, 1996), 189-195. 

45 See note 49. 
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system usability. In the 1996 book, Usability Evaluation in Industry, the SUS developer 
eontends that the most effeetive questions utilizing a Likert seale are those whieh solicit 
the most extreme responses: 

The items leading to the most extreme responses from the original pool [of 
50 potential questions] were then selected.. .items were selected so that the 
common response to half of them was strong agreement, and to the other 
half, strong disagreement. This was done in order to prevent response 
biases caused by respondents not having to think about each statement; by 
alternating positive and negative items, the respondent has to read each 
statement and make an effort to think whether they agree or disagree with 
it.46 


Due to the alternating positive and negative items, the SUS requires a 
scoring technique that accounts for the resulting numerical variance in raw scores.47 

Results from the SUS are presented in terms of percentages with 100% 
indicating perfect usability. As stated, this SUS scoring method is the standard manner 
described by the test’s designer to give an overall assessment of usability. The SUS 
questionnaire is provided in Appendix A. 

In addition to the SUS questions, participants were asked to respond to 
three additional statements using the same five point Likert scale. The responses were 
evaluated in relation to specific research issues of effectiveness and were not included in 
the SUS score calculation. The additional statements were: 

• I found the system supports my work 

• I was fairly confident that the information was the most current available 

• I was fairly confident that I found all the information by using this system 


46 See note 49. 

47 First, raw seores are eonverted to a SUS eontribution seore for eaeh individual question. SUS 
eontributions are ealeulated differently depending on whether the question is worded positively or 
negatively. Questions in whieh positive answers generate high seale ratings were seored by subtraeting one 
from the seale rating marked. Questions in whieh positive answers generate low seale ratings were seored 
by subtraeting the seale rating marked from five. This method gives equal eontribution opportunity for 
eaeh question with a range of zero to four and thus equates to a total forty possible points from the ten 
question survey. The total eontribution, then, is multiplied by two point five in order to put the SUS seore 
on a 100 point seale. 
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2 . 


Think Aloud 


Think-aloud protocol is widely accepted for the eolleetion of verbal data in a 
variety of disciplines ineluding eartography, engineering and reading eomprehension.^s 
This type of protoeol ealls for partieipants to verbalize their thought proeesses while 
eondueting a task, whieh a researeher ean later analyze to gain insight into a user’s 
cognitive proeess. Researehers Eriesson and Simon are aceredited with the core research 
for this type of verbal data in their 1980 artiele “Verbal Reports as Data.”49 

Usability studies often rely on this method for the collection of data. Respected 
usability praetitioner, Joe Dumas, indieated in a reeent review of the praetiee that, “Think 
aloud is the most important method we have in the toolkit of usability evaluation. It 
uneovers more problems than any other measure. 

There is some debate in the field, however, over appropriate use of the protoeol, 
and the validity of data, if it is not administered earefully. In a 2000 review of the 
method, authors Boren and Ramey indieate that praetitioners ean avoid these pitfalls by 
earefully adhering to four basie prineiples of the original researeh by Eriesson and 
Simon; 

• Partieipant introspeetion, inferenee or opinion must not be valued or 
actively elieited 

• Give detailed instruetions for thinking aloud 

• Remind partieipants to think aloud 

• Otherwise, do not intervene^ i 

Judith Ramey and others, “Does think aloud work?: how do we know?” In CHI '06 Extended 
Abstracts on Human Factors in Computing Systems (Montreal, Quebec, Canada, April 22 - 27, 2006). CHI 
'06. ACM, New York, NY, 45-48. DOI= http://doi.acm.org/10.1145/1125451.1125464 

K. Anders Ericsson and Herbert A. Simon, “Verbal Reports as Data”, Psychological Review, 87, 
no. 3 (1980): 215. 

See note 48. 

Ted Boren and Judith Ramey, "Thinking Aloud: Reconciling Theory and Practice," IEEE 
Transactions on Professional Communication, 43, no. 3 (2000): 261. 
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These prineiples guided the protoeol applieation for this usability test. 
Partieipants were told about the think aloud protoeol and given a detailed deseription of 
the proeess prior to eonducting the seenarios. I further described the process by 
conducting a practical example with each participant. Using a website, I described my 
actions while navigating the site. For example, I might state, “I’m looking for 
information on shopping so I think I will click on this shopping button...Oh, that didn’t 
work like I thought. OK I’ll try....” Participants were then asked to confirm their 
understanding of thinking aloud. 

While participants conducted tasks, I did not intervene in any way. Participants 
were, however, reminded to continue thinking aloud when they discontinued doing so. I 
employed a neutral, standard prompt, asking participants, “Can you tell me what you are 
doing now?” This method was employed in order to solicit continued verbalization 
without interjecting bias. 

The think aloud sessions were video recorded and I later transcribed observations 
and participant statements for trend analysis. Where trends were found, selected verbal 
data from the think aloud process were combined with post-completion interview data for 
incorporation into the analysis and findings. 

3. Qualitative, Semi-structured Interview 

Upon completion of the scenarios, participants were asked to discuss what they 
had experienced while working with the system. I maintained a neutral approach to 
interviewing in order to reduce potential bias. I accomplished the neutrality by refraining 
from leading questions which may have driven participants toward a particular response. 

The interview period began with the question, “Can you talk about your 

experience with the system?” Follow up questions were asked in reference to 

participants’ specific actions while completing the task. These follow up questions 

focused on suspected problem areas the participant had experienced during the test. 

However, these questions were presented in an open-ended syntax, also to avoid bias and 

solicit richer responses. For example, I would state, “While you were reading, you 

commented, ‘I need to go back up a little,’ could you discuss that a bit?” Furthermore, I 
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avoided using affirmations that might suggest agreement or otherwise influence a 
participants’ thought process. The interviewer typically acknowledged statements by 
using non-evaluative remarks such as, “OK,” or “mmmhmm,” and then continued to the 
next area of interest. 
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III. ANALYSIS OF RESULTS 


In this chapter, I discuss the findings from my analysis of trends in the user data. 
The findings relate to three areas: using the system in general, locating documents, and 
using documents themselves. Verbal analysis was conducted in three stages. First, the 
video recordings were reviewed to get an overall sense of participant actions and 
commentary. Second, the videos were reviewed again and activity and commentary were 
transcribed. Third, the transcriptions were analyzed to isolate potential themes. Last, the 
transcriptions were reviewed again to refine the data and ensure that the emergent themes 
held constant. 

Overall, I found that users felt the system is adequate and supports their work, yet 
search limitations as well as incomplete and inconsistent document libraries erode their 
confidence in using it. The data also show that users were not able to complete all tasks 
within the time limit by using this system. Further, it was evident that search and 
navigation deficiencies were a contributing factor. 

Additionally, I found that users are seeking some capabilities that do not exist in 
the current system. Specifically, users desire a capability to make documents 
personalized and portable so that they can easily access documents they use repeatedly, 
and users want to make notes or annotate areas within documents that they commonly 
use. Each finding is presented in a detailed data/discussion format which follows. 

A. USERS RANKED THE EXPERIENCE AS MODERATELY EASY 

The data show that users believe that the system supports their work and is not 
prohibitively difficult to use. However, usability rankings, user comments, and 
observation of problem areas indicate that users had a moderate level of difficulty overall. 
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1 . 


User Data 


My survey data indicate that users found the system moderately easy to use: 


• Participants ranked the overall usability (SUS) 71.5% on a 0%-100% 
scale 

• “I found the system supports my work.” (Mean response of 4.0 on a 5 
point scale) 


Users commented positively that the Marine Corps website (www.usmc.mil) 
assists their efforts in locating information: 


• “I knew where to go....I’ve used that [www.usmc.mil] before...If you 
click on publications it brings you to everything, forms, orders, 
directives.. .That’s not hard to navigate” 

• “I know immediately to go to the Marine Corps website....It breaks it 
down pretty clearly how you want to look them up...If I’m able to do 
keyword searches it saves a lot of time” 


Users expressed some difficulty with locating information on the site: 


• “It’s all right there if you can find it.... I usually use Google...once I 
figure out what I’m looking for then I can go to [www.jusmc.mil because 
it has the information out there” 

• “I didn’t feel it was bad....Once I found what I was looking for.I had 

to search around a little bit... .1 didn’t think it was bad” 

• “.. .Not sure where it is on here.. .I’m sure it’s on here somewhere.” 
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And users often left the site to conduct keyword searches: 


• 5 of 5 participants abandoned the site to use commercially available tools 

(Google) when searching for keywords (2 after failed attempts to first use 
the keyword search on the site) 

2. Discussion 

Results from the SUS questionnaire and users’ general reaction to using the 
system were relatively positive. The SUS score (71.5%) and the subsequent user 
comments suggest that users are tolerant and accepting of the current system which, as 
indicated by the positive assessment, does help support their work (mean score 4.0). In 
large part, this positive attitude seems to be centered on the idea that users felt confident 
that the documents could be found from the official USMC website, citing, “7 knew 
where to go, ” and “it has the information out there." 

Some aspects of the user comments may indicate that while the overall experience 
was positive, elements affecting its ease of use may need improvement. For example, 
when interviewed, users inserted qualifiers such as “it’s all right there if you can find it," 
“7 had to search around a little bit." and “I’m sure it’s on here somewhere. ” Yet another 
user pointed out the site’s limited ability to help determine which documents were needed 
to accomplish a task, “I usually use Google. Once I figure out what I’m looking for, then 
I can go to usmc.mil.” 

Another indication of user difficulty was evident because users often abandoned 
looking for documents from the USMC site after encountering unexpected or blank 
search results or anytime a keyword search needed to be conducted. The use of qualified 
statements and abandonment of the site indicates that while users expressed overall 
optimism about the system, some elements of effectiveness and efficiency reduce the 
users’ overall satisfaction with using it. 
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B. USERS SEEK REASSURANCE THAT INFORMATION IS CURRENT 

AND COMPLETE 

The survey results and user eomments also indicated that users may lack 
confidence that the system will provide complete and current information, although some 
positive data exist in this area as well. Users tended to indicate that they thought the 
information would be current, but expressed some concern as to whether it actually was, 
in fact, current. 

I. User Data 

Collectively, users responded neutrally in the post-test survey when asked to 
report on their confidence in finding all of the salient information by using the system: 


• “I was fairly confident that I found all the information by using this 
system.” (Mean response of 3.0 on a 5 point scale) 


User comments further identified a lack of confidence that simply retrieving 
information online will provide complete and reliable answers to their work-related 
questions: 

• “I choose the paper version because there is a choice.I think that bleeds 

over into the confidence in the documents.” 

• “.. .1 would probably take more time to be sure of that [information found] 
and bounce it off what I find in other documents before I confirm 
anything.” 

• “I would have gone to the chief to make sure... [What are you doing 
now?] Going back through to double check... If I’m going to talk like an 
expert I wanted to reread it” 
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Post-test survey data were more positive in regard to the eurreney of information, 
but still indieates mild reservations: 


• “I was fairly eonfident that the information was the most current 
available.” (Mean response of 3.6 on a 5 point scale) 


User comments generally supported that they thought information would be 
current due to the fact that it is electronically updated and maintained, but they were not 
assured of this currency, which they felt to be important: 


• “Maybe it’s just blind faith, but it seems like it’s updated pretty 
regularly....the USMC site...I don’t know if it’s just trust....I never felt 
like I would get the wrong thing... .1 have confidence.” 

• “You’d want to make sure it’s up to date. this type of information 

changes on a yearly basis....I feel that having the most up to date is 
important... .electronic documents are going to be the most up to date.” 

• “I didn’t check the date but I have to assume through the USMC site 
they’ll publish the most up to date stuff.” 

• “...the last time I was doing supply stuff, it wasn’t really updated. I 
thought about going here [www.usmc.mil]....but didn’t because it wasn’t 
a real positive experience.” 

• “I don’t know how old this document is... Maybe it’s updated more often 
than this... [referring to RCO site]...I would assume these guys are the 
head.. .they’ve got the [most current info].” 

• “This document is not up to date.. .it’s talking about [limit of $2500] when 
I know its up to $3000.. .But I’ll use it nonetheless... that would make me 
a little nervous that information I take out of here may not be entirely 

accurate .... I would follow up with a call to the contracting office.” 

39 




• “I feel this is eurrent and up-to-date... [www.Jusme.mil everything is up- 
to-date. The latest and greatest.... I feel more eonfident on here than what 
I would find in a pub binder [printed].” 

2. Discussion 

As partieipant four aeeurately points out, trust is a key faetor in using information. 
Policy providers therefore need to ensure that their information supports ''confidence in 
the documents” and "having the most up-to-date” information. Yet users found 
information which 'may not be entirely accurate,' 'maybe...updated more than this,' and 
were uncertain as to 'how old this document is.' These comments indicate some concern 
exists over currency of information, particularly during scenario two, when users 
retrieved information from locations other than www.usmc.mil. Overall, however, users 
indicated that they felt www.usmc.mil provided 'the latest and greatest and they 
expressed confidence through a marginally positive survey response (3.6). This marginal 
confidence is apparently representative of user reliance on 'blind faith,' and "assuming” 
that the sites are up-to-date, rather than an explicit understanding that the information is 
current. 

Another aspect of trust is the completeness of the information. Users indicated in 
the survey that they somewhat lacked confidence that all information had been located 
(mean response 3.0). This shortcoming is evidenced in user comments indicating they 
would ‘choose paper,’ ‘bounce off other documents,’ or consult other sources to ensure 
reliability and completeness of information. 

C. USERS WERE UNSUCCESSFUL IN LOCATING POLICY DOCUMENTS 

FOR A MARINE-CORPS- SPECIFIC TASK 

Although I have noted that users indicated knowledge of the www.usmc.mil web 
site as a primary source, users experienced difficulty using the site to locate the document 
required to complete a Marine-Corps-specific task. This was evidently the result of 
inadequate search functionality and a mismatch between the site layout and the way users 
view the content. Both items will be discussed further in later sections. 
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1 . 


User Data 


Users had difficulty locating documents within the USMC web site: 


• 4 of 5 participants failed to locate the policy document required to 
complete the property control scenario within the time limit. 

Most users did not use the site search function, and those that did were 
unsuccessful: 

• 5 of 5 participants used www.usmc.mil in combination with Google as 
their primary search mechanism 

• 2 of 2 participants failed when using the www.usmc.mil search box as a 
primary method for searching 

• 3 of 5 participants did not attempt to use the www.usmc.mil search box 


Only one user was able to find a document through site navigation: 


• 2 of 3 participants failed when using general site navigation as a primary 
method augmented by Google keyword searches 

• 1 participant succeeded by using navigation, choosing the option to 
browse ‘all MCOs by SSIC’ 

2. Discussion 

Users attempted to retrieve documents by navigating within the site (3 users), by 
using the site’s internal search capability (2 users), and by augmenting their search with 
Google.com (5 users). All three methods proved to be ineffective, resulting in the high 
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failure rate (4 of 5 users). The one user that was suecessful navigated to the doeument 
and loeated it by its document number. The other four users, who did not know or search 
by document number, were unsuccessful. This indicates two things: first, that document 
number knowledge is useful for searching, but second, most users do not search in this 
way. The other four users searched instead using key terms drawn from the task, such as 
“CMR,” “SL-3” and “accountability.” In subsequent sections, the discussion of specific 
search and navigation difficulties provides additional observations, which may clarify the 
users’ view of the task and problem space. 

D. USERS WERE MODERATELY SUCCESSFUL IN LOCATING POLICY 

DOCUMENTS FOR A NON-MARINE-CORPS-SPECIFIC TASK 

Users had greater success with a wider topic area by employing broad internet 
search capability via Google. All users were guided by the awareness of a potential 
source for the information (Regional Contracting Office (RCO)). However, users did not 
seem to know how to get to the source directly. Only one user relied on the 
www.usmc.mil site, and that user was unsuccessful. 

I. User Data 

Users experienced greater success in locating a document that was not specific to 
the Marine Corps: 


• 3 of 5 participants succeeded in locating a policy document required to 

complete the financial management scenario within the time limit 
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The increased success was not attributable to www.usmc.mil. In fact, users either 


didn’t consider it as an option or failed by using it: 


• 2 participants never considered the site 

• “I never think to look on USMC site for credit card info....1 didn’t 
even look on USMC site...it could very well be there” (participant 
succeeded using Google) 

• “Is it the FAR?.. .I’ll try DoD.. .Which agency would be in charge 
of that.. .I’ll go to DAU...” (participant failed using DAU) 

• 1 participant failed by attempting to navigate to the RCO site through 
www.usmc.mil) 


Successful users primarily used broad internet search capability (Google) and user 
knowledge of a potentially viable source (Regional Contracting Office (RCO)). One user 
found the site immediately: 


• 1 used Google to search for the RCO and then navigated to a card holder 

(CH) desk guide on the site 

However, two other users were either ‘lucky’ or used a back-door method to find 
a source: 


• 1 participant used Google to search for ‘regional contracting southwest,’ 

then opened a PowerPoint file returned by the search, then identified the 
RCO URL embedded in the Power Point, and only then retrieved the 
Approving Official (AO) desk guide. 

• The participant commented, “The USMC publications...it’s just 
hard to search if you don’t know the exact title or number, which I 
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will usually try to find it through Google or my paper 
doeuments...” 

• 1 participant used Google to search for ‘government commercial purchase 

card,’ then followed a link to the HQMC GCPC SOP. 

• The participant commented, “I could have searched all day and not 

found it.I just got lucky that exactly what I want popped up... .1 

could have gone back to USMC... .1 don’t know exactly where it is 
on there.... I guess I got lucky really.... I just did a Google 
search...and I got lucky a current SOP came up when I clicked on 
the link” 

2. Discussion 

Interestingly, only one user attempted to retrieve a policy document for this 
scenario by using the www.usmc.mil website and that user was unsuccessful. Of the 
three users that were successful in attaining a valid document, none used the USMC site. 
One used the HQMC GCPC SOP retrieved via Google, and two used GCPC desk guides 
retrieved through the RCO web site, also located via Google. These methods, combined 
with user comments, indicate that while the policy information is available from within 
the Marine Corps, users are not confident on where to obtain it:, “7 could have gone back 
to USMC...I don’t know exactly where it is on there...I guess I got lucky,” or “7 never 
think to look on the USMC site for credit card info, ” and 'dt’s just hard to search if you 
don’t know the exact title or number. ” Users, then, resort to internet searching, despite 
the designation of www.usmc.mil as the official site for policy within the Marine Corps. 
One user even resorted to a back-door method, locating the website from a PowerPoint 
file loaded on a server. While users should have options available to them and may still 
choose to search for policy in familiar ways (like Google), the utilization of 
www.usmc.mil may prove more effective if all available policy guidance were current 
and available there. 
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E. LACK OF A SINGLE, REPUTABLE, CONSOLIDATED SOURCE 

CONTRIBUTED TO INEFFICIENT SEARCH AND NAVIGATION 

Users, on average, used between three and six pathways in their attempts to loeate 
documents. Users that did not know the source document by title or numerical identifier 
had difficulty determining the location of the document. Users commented that a ‘one 
stop shop’ would be a preferred addition to the current organization to help alleviate this 
problem. 

I. User Data 

Users commonly resorted to inefficient methods of locating documents: 


• An average of 6 pathways were used to locate documents for scenario one: 

• Units => by location => North Carolina => MCCSSS; 

• Publications=>Marine Corps Publications=>PLMS; 

• Publications=>Marine Corps Publications=>Technical Manuals; 

• Publications=>Orders/Directives=>Misc Pubs; 

• Scarehbox using‘Orders’fdter; 

• Searchbox using ‘Directives’ filter 

• Publications (from the drop menu - clicking directly as if it were a 
link, when it is not) 


• An average of 3 pathways were used to locate documents for scenario two: 

• Google=>DAU web site: 

• Units=>Alphabetically=>Camp Pendleton (looking for contracting 
office); 
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• Publications (from the drop menu - clicking directly as if it were a 
link, when it is not) 

• Google=>(in search of RCO web site) 


Only one user knew the doeument number and was suceessful in navigating the 
USMC web site to find the doeument, other users were unsuccessful on the site: 

• 1 of 6 navigational paths followed on www.usmc.mil resulted in the 
suecessful location of a document [browse by SSIC (partieipant knew the 
document number)]. 

• The suecessful partieipant commented, “I think it’s aetually 
organized really well right now....you do have the options and it’s 
all very clear, straightforward.” 

• 5 of 6 navigational paths followed on www.usmc.mil were unsuceessful. 
Paths included Publication Library Management System (PLMS), 
miseellaneous pubs, Teehnieal Manuals, and others. 

Users further eommented on the need to know doeument identifiers: 

• “I’m looking for the order/pub, for the one that lists the TAMs....I don’t 
know what that is...[checks paper doeument, goes to Google]...the other 
thing is, just eall division.” 

• “I’m going on a wild goose chase I have no idea...Just googling random 
[things] to see what I can find... I don’t kn ow the name of the pubs.” 
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Users also expressed that a eonsolidated listing of all doeuments would help meet 
their needs: 

• “I prefer a reputable one-stop shop...maintained with semblanee of 
authority” 

• [If you could organize how you wanted?]... “Listed by the SSIC code^2 
regardless of whether it’s a user manual, or P-directive^3 or 
category....listing all by code it might be easier like if you didn’t know 

exactly what you were looking for.not to get rid of everything else but 

just another option.so it’s geared toward each person’s preference” 

• “I thought there was a way you could just list them all here,” [and later, 
after two failed site searches and attempting to use the ‘Publications’ 
menu option]... 'Is this the way you get the entire listing?' 

• “....I thi nk my only problem is not all the pubs are on there 
[www.usmc.mil]...What I really want is the [user manual#] 124 and I 
don’t know where to get it.. .I’ve looked for some pubs that were not listed 
there [www.usmc.mil] and I’m not sure why.” 

2. Discussion 

In looking for policy documents, users did not seem to have an efficient means 
through which they could locate the document desired, when looking for Marine Corps 
specific documents (average of 6 methods used). The only successful navigation method 
on the site was to seek a document by its numerical identifier (1 of 6 site navigation 
attempts), yet most users did not know the document identifier. For example, “I’m 

SSIC is the Standard Subject Identification Code, a taxonomy used throughout Naval 
documentation to organize documents by operational category. SSIC is itself controlled by regulation. See 
Department of the Navy SECNAV M-5210.2, Standard Subject Identification Code (SSIC) Manual 
(December 2005). 

Under the current organization, P-directives are listed separately and do not fall in numerical 
sequence with other directives and documents which are not preceded by a ‘P’ in the document number. 
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looking for the order/pub, for the one that lists the TAMs..../ don’t know what that is” 
and, “I’m going on a wild goose chase.../ don’t know the name of the pubs.” Thus, 
document identifiers alone do not support the manner in which users look for 
information. 

Users sought documents in varying ways based on the information from the 
scenario or other key identifying information relative to the task. However, users did not 
always have a clear concept of where that information might be located: “I thought there 
was a way you could just list them all here. ” And “I’ve looked for some pubs that aren’t 
listed there, and I’m not sure why.” Users also requested a complete, consolidated view 
of the information that is available, stating, “I prefer a one stop shop from a reputable 
source. ” These findings are consistent with usability best practices which suggest that 
sites should maintain multiple methods of entry to information and be consistent with the 
ways people do their work.^^ 

F. USERS LACKED CONFIDENCE THAT THE USMC WEBSITE 
(WWW.USMC.MIL) SEARCH ENGINE WOULD PROVIDE ADEQUATE 
RESULTS 

Users verbally stated their lack of confidence in the USMC search tool. Users 
that attempted to use the search tool found no relevant results. As a result, most users 
frequently relied on Google to conduct searches when looking for policy documents, 
despite commonly acknowledging that the www.usmc.mil site is the primary resource for 
policy documents. 


U.S. Department of Health and Human Services, Research-Based Web Design & Usability 
Guidelines, Chapter two. http://www.usabilitv.gov/pdfs/guidelines.html#2 (accessed December 1, 2007). 
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1 . 


User Data 


Users generally expressed a lack of confidence that the search box on 
www.usmc.mil would be helpful: 


• “You have to know exactly what you’re looking for....It’s hard to 
search....the search engine on the website...! haven’t had much success 
with.. .1 usually will go to Google or look in my [paper-based] pubs.” 

• “You generally don’t use [the www.usmc.mil search tool] because it’s 
never really done what I wanted...To me, as a user, it seems like the 
search engine is kind of weak.” 

• “The times I have used the search box hasn’t brought up what I 
want... .it’s not the most user friendly...” 

• [What would make your experience better?] “Maybe a Google-type search 

engine on usmc.mil where it brings up kind of what I’m looking for.a 

stronger, more user friendly search engine on the USMC website...” 


This lack of confidence led users to rely primarily on Google for searching: 


• 5 of 5 participants eventually abandoned the www.usmc.mil site for 
Google when attempting to search by keyword 

• 2 of 5 participants attempted the www.usmc.mil search box first, and then 
immediately abandoned the site for Google after getting zero results 
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Users that attempted to use the seareh eapability were eonfused by the seareh 
filters, and abandoned it due to laek of seareh results: 


• 1 participant tried to search without a keyword [seeking a full list of 
publications], and then with the keyword ‘supply manual’ using the orders 
filter and then again by using the directives filter - all three returned zero 
results 

• The participant commented [after zero search results], “I’d stick 
with Google at this point.. .I’m getting something” 

• And later, during the interview, “I was looking for orders and the 
dropdown menu reset to directives... until you get results it’s not 
clear .. .did it search for orders or did it search for directives?” 

• 1 participant attempted to search for ‘accountability,’ first using the orders 
filter, then again with the directives filter. Both returned zero results.. .the 
user abandoned the site for Google 

2. Discussion 


Users avoided using the search tool on the www.usmc.mil website, indicating that 
they '^haven’t had much success with [it],” 'Ht’s never really done what I wanted,” or 
“it ’5 not the most user-friendly. ” All of the users in the study primarily used Google to 
search, presumably in part due to this impression of the site’s 'kind of weak’ search 
capability. 


The search capability of the website should be easy 
to use and provide expected results. In this study, the 
search tool was deficient in both areas. First, users did not 
find the tool easy to use. The two users that chose to use 
the tool both had difficulty with the search filter, first 
applying the ‘orders’ fdter, and then, after no results. 


Search 


Search for the following word(s): 


Choose section to search: 


Directives 


Search 


Advanced Search 


Figure 1. USMC Search 
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applying the ‘directives’ filter, which also returned zero results for their search terms (See 
Figure 2). One user noted that regardless of the filter applied, the tool defaulted back to 
‘directives’ after the search was complete and he commented, “/t’5 not clear...did it 
search for orders or did it search for directives?” Additionally, since both users 
attempted the search for their keyword using each filter successively, it would appear that 
users do not understand what areas of the site correspond to the different search filter 
selections ‘orders’ and ‘directives.’ detracting from the search tool’s ease of use. These 
data suggest that the search filter may need improvement to alleviate user confusion. 

Second, the search tool did not provide expected results. The users conducted the 
searches using terms they suspected to be in the target document title or text. On each 
occasion that the search tool was used, zero results were returned (both users, either 
filter). The search term "accountability,’’ for example, is prevalent throughout the target 
document, yet the search engine did not return any results for this search term. The term 
"supply manual’ should have returned the intended source document (from the title - 
Marine Corps Consumer Level Supply Policy Manual) yet it did not. These examples 
highlight that the search engine does not perform the way users expect and causes users 
to leave the site. As one user indicated, ""I’d stick with Google at this point. I’m getting 
something. ” 

G. USERS NEED BETTER WAYS TO ACCESS AND EASILY RETURN TO 

THEIR FREQUENTLY USED DOCUMENTS 

Some users indicated the repetitive nature of their research and expressed a desire 
to maintain availability of their commonly used documents without searching again or 
downloading them each time. 

I. User Data 

Users repeatedly use the same source documents for information; 


• “There’s like two pubs used for 99% of stuff we did... for the majority of 
answers you go through.. .you go to the same stuff over and over again.” 
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Additionally, users seek ways to globally aeeess those doeuments: 


• “I’ll use Google or USMC [www.usmo.mil] beoause I may not be at my 
oomputer.. .1 won’t have my dooument with me.... Maybe [if it brought up 
my desktop...] that way I oan referenoe it.” 


Without this feature available, users attempt to create portability in their own 

way: 


• “I usually use my own stuff...” [referring to paper documents] 

• “Normally, a lot of this stuff is bookmarked” 


The nature of the military environment also reinforces the need for global access: 


• “When I was at my old unit, I actually had that downloaded to my desktop 
[while observer retrieved the primary source doc]... I didn’t want to take 
time to download everything.” 

2. Discussion 

In addition to the ‘one-stop shop’ previously identified, users are concerned about 
document organization and availability as it relates to their common needs. As one user 
commented, “You go to the same stuff over and over again.” Naturally, then, users do 
not want to repeatedly search for the same documents. These data imply that efficiency 
may be increased by giving users an option to customize a policy screen, which would 
allow them to electronically create the personalized, portable access desired and “use 
their own stuff’. Another option might be a ‘frequently used’ component for the 
interface where people can quickly see the most frequent items used. 
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H. USERS WERE NOT SUCCESSFUL IN QUICKLY LOCATING ALL THE 

INFORMATION NEEDED FOR DECISION-MAKING 

The data show that users who referred to proeedural doeuments had more success 
in locating information to answer questions than those that strictly used policy 
documents. Those users who were seeking procedural information in policy documents, 
(a common occurrence as we have shown) were unsuccessful in locating answers. 

I. User Data 

In scenario one, most users found partial answers but were not able to answer the 
Commander’s question regarding how to properly account for UURI items: 

• 4 of 5 participants correctly identified the Commanding Officer as the 
proper authority to establish allowances 

• 5 of 5 participants consulted MCO P4400.150E (either located or provided 
to them) as the source document for UURI 

• 0 of 5 participants located the information on how to account for UURI 
component items within this policy document 


In scenario two some users found enough information to advise the Commander 
against the purchase by using guides and SOPs, but none found all the information 
requested: 

• 0 of 5 participants located all information requested 

• 3 of 5 participants determined that the purchase was unauthorized by using 
desk guides or the HQMC SOP (procedural documents) as a source 
document 

• 1 of the 3 participants commented, “I go to CH desk guide because 

it has ‘in the weeds’... .answers” 
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Users provided the policy document NAVSUPINST 4200.99, however, were 
unsuccessful in making a determination: 


• 2 of 5 participants were provided NAVSUPINST 4200.99 as a source 

document 

• 1 of the 2 participant prematurely advised that their was ‘a 
possibility’ that the purchase could be authorized under the 
necessary expense rule (NAVSUPINST 4200.99 as source) 

• I of the 2 participants made no advisement (time expired) 

2. Discussion 

Users who referred to policy documents (MCO 4400.150E for scenario one and 
NAVSUPINST 4200.99 for scenario two were unsuccessful in finding all required 
information to complete the task. Users who referred to procedural documents like desk 
guides (2 users) or SOPs (1 user) were able to find the information that enabled them to 
accurately advise the Commander in scenario two. Yet, the tasks required users to cite 
policy on the subject, not to describe the procedures they would use. This indicates that 
policy and procedural documents may not be entirely severable, suggesting a need for a 
more comprehensive, integrated system of information availability. In fact, users seemed 
to prefer to use procedural documents to find policy answers (3 of 5 users). As one 
successful user indicated, “7 go to the desk guide because it has ‘in the weeds ’ answers. ” 

As previously evidenced from the survey results and validation of the scenarios, 
the user group is typically driven to policy documents to solve procedural questions to 
ensure that procedures are in accordance with policy. Since policy documents are not 
necessarily organized by procedure, users may be more inclined, and more successful, 
when using procedural documents in place of the source policy document because they 
reflect the user’s goals more directly. 
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1. USERS HAD DIFFICULTY MOVING WITHIN DOCUMENTS 

Users spent a lot of time serolling within doeuments to locate information referred 
to from other references within the document (for example, the Table of Contents (TOC) 
and chapter headings). To circumvent the problem, users initiated work-arounds, but 
suggested enhancing documents with better search capability and internal hyperlinking. 

I. User Data 

Users unsuccessfully attempted to locate information within documents using 
keyword searches. 

• 3 of 5 participants attempted to search for a keyword, but moved to the 

TOC after receiving ineffective results 


Users scrolled between the TOC, the Index, and Headings to locate the 
information they needed. 


• 4 of 5 participants used the TOC and subordinate headings extensively to 

narrow their search: 

• “I was looking at TOC for a keyword....which chapter, how’s it 
going to break it down, what keywords do THEY use...[then, 
reading chapter headings]...It’s talking about principal end 
items...I know I need to read through 1005-1007, 1008 too.” 

• [After three failed keyword searches].. .“I’m going to go to TOC in 
this case....which is clearer.” 

• [reading TOC headings] “TAM authorized materials that are where 
I’m going to go.” 
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• “....I could get that [keyword (SL-3)] anywhere so I went to the 
index to try and narrow it down a lot quieker.” 

• “The doeument was hard to negotiate...[due to laek of ehapter 
headings]” 

• 5 of 5 partieipants scrolled repeatedly between doeument loeations when 

they had already determined the loeation they sought from a TOC or 
ehapter heading; 

• [user finds TOC and then pages down to ehapter 5, ‘aeeountability 
proeedures’...then reads ehapter headings] “...I might have been 
wrong... Chap ter 5 is what I thought I wanted but maybe not.” 
[serolls baek to TOC] 

• [Serolls from TOC through to index...TOC led him to inventory 
eontrol prooedures....in index he found ‘SL-3’....] “It’s different 
chapters....so I’m going to go here...” [serolls baek up to find the 
loeation] 


Users had to rely on memory and workarounds for moving between referenees: 


• “It refereneed another paragraph...I’m going to go to that...should've 
wrote that down...I’m going to seareh that.” [eopies paragraph number 
and pastes into seareh tool] 

• [partieipant types TOC refereneed page number in the search tool to locate 
the page] “...The page numbers are on the bottom, but the forward button 
takes me to top of page [so I ean’t see the page when I use it]...It’s 
delaying the proeess.” 
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Users identified more efficient ways to retrieve information. 


• “I think I could go fas ter.... with a paper document... the way I do 

things.I flip to the front, I flip to the back, I look for some keywords.” 

• I really like [keyword searching]...that way I know immediately if what 
I’m looking for is in that pub or I have to scroll through it.. .before I knew 
to do that I used to use the TOC and it would take me a lot longer.” 

• “I do like the ones where the TOC is linked.” 

• “The only other thing is....hyperlinking....for example....! love when 
things are hyperlinked...” [Refers to internal document reference from 
paragraph 2011 to paragraph 2003 (but no hyperlink)] 

2. Discussion 

Primarily, users were navigating Adobe pdf files that had produced in a manner 
which allows keyword searching but mostly had no embedded hyperlinking (except the 
HQMC SOP which was created in Microsoft Word format). From the user data, it is 
clear that better navigational aids would increase efficiency. As one user indicated, “7 
love when things are hyperlinked. ” 

Most users (4 of 5) relied heavily on the TOC and subordinate chapter headings to 
narrow their search. However, the lack of internal hyperlinking or other navigational aids 
resulted in inefficient navigation when moving between headings and topic areas. The 
lack of navigational aids caused users to scroll a lot, take notes to remember document 
areas, and cut-and-paste words into the search tool to navigate internally. One user 
attempted to page through with the page buttons, but found that it was ""delaying the 
process. ” All of these methods proved to be inefficient as users navigated from topical 
headings to document text and back to topical headings. One user put it simply, stating, 
“7 could go faster with a paper document.''’ 
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J. USERS NEED RICH CONTEXT TO UNDERSTAND SEARCH RESUUTS 

AND IDENTIFY KEY INFORMATION 

Several users eommented that keyword searehing, while benefieial, usually takes 
them direetly to a plaee in a doeument that does not allow them to see the greater eontext 
in whieh the finding is presented. This ereates ineffieieney for users who then attempt to 
reveal the eontext by scrolling to the topical heading or by other means. 

I. User Data 

Four users commented on the need for topical headings to be associated with 
keyword findings: 


• “Sometimes the greater section heading let’s you know what context you 
are looking at that paragraph in....sometimes just going to the instance 
doesn’t give you the answer.” 

• “If I went right to it [with keyword search] I might have to go back up 
and re-read so I just decided to read the whole thing” [Re-read?] “Yeah, 
to get the background... if you go right to that exact term you might have 
missed some key information.” 

• [You said, “well, I need to go up a little bit”] “I wanted to double check to 
see if I was in the right area.” 

• [On finding a keyword, then scrolling up] “to see what are we talking 
about here....what chapter are we in...” [scrolls up to see] 

2. Discussion 

Users often felt the need to “double check to see if I was in the right area. ” Users 
found the double-checking inefficient but also necessary to determine “what context you 
are looking at” or to see “what chapter are we in.’’’’ 
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Users are looking for contextual information to help them determine whether they 
have ''missed some key information, ” or "to see what are we talking about here. ” When 
the search tool would drive users to the exact location of the keyword, with no contextual 
information about the larger topic area, users were then concerned that it may be the 
wrong area, or at least needed to understand where they were within the document 
structure overall. 

User comments also indicate that seeing information in context helps them 
accomplish their goals more effectively and efficiently. Users want to “get the 
background” and “to double check”. Research shows that it is important to layer 
information so that users who wish to can access rich contextual information.55 
Furthermore, context is beneficial when looking for the answers to a more complex 
scenario where there is more than one step to a process or the user goal is equally 
complex. 

K. USERS LOOKED FOR INCREASED SEARCH FUNCTIONALITY 

Although users primarily resorted to navigating documents through the TOC and 
subheadings, several attempts to conduct keyword searches were unsuccessful. The 
frustration users experienced were manifested in their interview responses, expressions 
during their search, and the resulting abandonment of the search function. 


Beverly B. Zimmermann, “Applying Tufte's principles of information design to creating effective 
Web sites.” In Proceedings of the 15th Annual international Conference on Computer Documentation (Salt 
Lake City, Utah, United States, October 19 - 22, 1997). SIGDOC '97. ACM, New York, NY, 309-317. 
DOU http://doi.acm.org/10.1145/263367.263406 
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1 . 


User Data 


User searches were ineffective: 


• 4 participants attempted a total of 17 keyword searches with only 1 
useable result in scenario 2 

• Some unsuccessful terms included ‘single-purchase,’ ‘prohibited items,’ 
and ‘vehicle repair’ and in scenario 1, ‘CMR’ 


Users were frustrated by search difficulties: 

• " ....I can’t type in ‘SL-3’ and ‘accounting’ because I don’t get 
anything...In adobe it will only find the search terms verbatim...It’d be 
neat to have that feature” [referring to ability to search for 
chapter/paragraph with multiple search terms] 

• [user reverts to search box and types ‘single-purchase’]...”! think it’s 
hyphenated... [no result]...nope...What if I type in purchase limit....ok 
it’s not hyphenated.” 

• [uses search box again, typing ‘except’...Fist on chin...hmmmmmm 
grunts] ’’...give me exceptions!” 

• “CTRL-F wasn’t really working... [uses keyword search for ‘threshold,’ 

then ‘limit’].none of those keywords are in there...I used a poor 

key word... should have used prohibited [not ‘unauthorized’].that’s 

why I just started scrolling.” 
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2 . 


Discussion 


Users experienced frustration with inadequate search results. Specifically, one 
limiting factor of the search capability is that it only returns verbatim matches when users 
expect a more robust search that responds to common key words and has some error 
correction. One user illustrated this, commenting while searching for purchase-limit, 
nope... What if I type in purchase limit.... ok it’s not hyphenated.” Clearly, the user 
expects that the search tool should be able to locate the term whether it’s hyphenated or 
not. Similarly, another user sought to find unauthorized items by using the keyword 
"unauthorized, ’ and upon no results, commented, “7 used a poor keyword...! should have 
used ‘prohibited, ’ that’s why I just started scrolling.” When confronted with poor search 
results, users reverted to other methods including scrolling and using a commercially 
available search tool that did provide effective search results (Google). 

Additionally, users did not achieve expected search results using terms from their 
task. One user discussed this limitation and his desire to use keywords that made sense 
from the task stating, “7 can’t type in ‘SL-3’ and ‘accounting’ because I don’t get 
anything.” A robust search tool that could provide ‘or’ search results, rather than 
searching strictly for the entire phrase would have brought this user directly to the ‘SL-3’ 
section where the answer was located. More advanced search tools even provide results 
based on relevance of the relation of keywords as they appear in proximity in the text. 
One user referred to this, noting, “It’d be neat to have that feature.” 

Search tools such as these increase the likelihood of finding the keyword in the 
appropriate context. Additionally, since users draw keywords from their tasks, they may 
resort to synonyms or related terms. A more advanced search engine would be capable of 
matching similar words and would have discovered matches for things like ‘prohibited 
items.’ This search feature may greatly assist users in finding documents. 

L. USERS WANT TO HOLD THEIR CURRENT PLACE WHILE 

CONTINUING TO CROSS-REFERENCE 

As I indicated in previous findings, users in most cases rely on the TOC and 

subheadings to narrow their search. Additionally, the documents have internal cross- 
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references that refer to other information or locations. In either case, if the information in 
the new location is not what users expect, or if, from the new location, they need to go 
back to the original location, they may not be able to find their way back. In short, the 
user requires a method to get back to the point of origin easily. 

1. User Data 

Users’ navigation behavior and comments demonstrate inefficient work-arounds 
to go back to previous information; 


• "I feel like I’m close but...It referenced another paragraph...I’m going to 
go to that...should've wrote that down.” [user scrolls between paragraph 
2011 and 2001] 

• Let me go back up to chapter three [to see what it’s about]...maybe I just 
didn’t use the right word...Let me go back to the top and look at my 
choices again.” 

• [Uses TOC and then scrolls to chapter 5...looking over chapter headings] 
“I might have been wrong... chap ter 5 is what I thought I wanted but 
maybe not.” [scrolls back to TOC.. .then chapter 2.. .then TOC again] 

2. Discussion 

The data reiterate user reliance of TOC and sub-headings to narrow their search. 
However, users commented, “I might have been wrong, ” or “Maybe I just didn’t use the 
right word” indicating that users don’t always select accurately from the title headings 
and may need to easily “go back to the top and look at choices again.” Additionally, 
users wanted to investigate internal cross-references but couldn’t easily do so because 
they didn’t have a clear method of returning to the current location. Users instead chose 
to write down references for future investigation, since the capability did not exist. 
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Enhancing the system to support this need would increase the effieieney of the 
information retrieval proeess. Designers may consider adding a “saved file” function or 
supporting internal bookmarking, otherwise known as anchor points, or anchoring. 

M. PERSONALIZATION/CUSTOMIZATION IS IMPORTANT TO USERS 

Users eommented that some information is referred to repeatedly, and that 
flexibility with paper-based doeuments allowed them to tab and highlight frequently used 
information. Users commented on the desire to replieate these features with eleetronic 
doeuments by providing editing tools that allow highlighting and annotating. 

I. User Data 

3 of 5 users indieated a desire for personalization features: 


• “I don’t know if you can make notes.if that was an option [id like it].” 

• [Why use paper based over website?] “It’s right there... It’s easier... I 
have everything tabbed....1 flip through I know where everything is...” 

• “You use it so mueh we just had it printed off... .we all had it highlighted 
and tabbed and marked....for the majority of answers you go 
through.. .you go to the same stuff over and over again.” 

• “....I have a hard eopy to flip through....highlight....for this partieular 
doeument I would be all about that.. .In my particular publication... .things 
that are key... .1 highlight in the publication.” 

2. Discussion 

Users stated that their work repeatedly drives them to the same poliey documents, 
and the same loeations within those poliey documents. As one user stated, “Tom go to the 
same stuff over and over again” Another commented on the frequency of returning to 
eommon doeuments, stating, “you use it so much. ” Naturally, users want to identify 

these eommon doeuments for easy retrieval later. 

63 




In the paper-based system, users were able to take advantage of the physical 
presence of the documents and could use tools which allowed them to identify the '"things 
that are key” within documents. Users typically "had [the document] highlighted and 
tabbed and marked. ” As the users indicated, the ability to personalize the documents 
made their work "easier."' Thus, users seek a capability to mimic their paper documents 
in order to replicate this advantage. Adding the capability to bookmark, highlight and 
annotate documents could prove to be a significant enhancement to the usability of the 
system. 
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IV. RECOMMENDATIONS AND CONCLUSION 


My analysis concluded that several actions could be taken to improve the usability 
of the current system. In this section, I outline those recommendations and discuss 
possible implementation considerations, limitations, and the need for further testing. 


A. RECOMMENDATIONS 

Based on my findings and applicable industry best practices, a consolidated listing 
of recommendations to improve the usability of the electronic supply policy document 
system is provided below, followed by an discussion of each recommendation in the 
sections that follow. 


• Provide a single point of entry to all documents and ensure content is 
current and complete 

• Improve the www.usmc.mil search engine 

• Provide users with both site and document customization options that 
support their work 

• Provide a collapsible structure for navigating and hyperlink internal 
references 

• Enhance documents with contextual aids 

• Improve internal document search capability by moving away from 
current document format 

• Consider implementing a more robust database 

• Conduct further research to help identify additional problem areas, 
underlying users’ needs, and best practices 
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1. Provide a Single Point of Entry to All Documents and Ensure Content 
is Current and Complete 

The document system will be more effective if it allows users to search for 
documents in ways that support their work. The www.usmc.mil website is an 
established, reputable authority for Marine Corps Directives. However, the document 
collection was incomplete and documents were not organized in ways that supported 
user’s mental models. This resulted in several users abandoning the site. 

When going to the USMC site, users were looking for a complete list of relevant 
documents. My first recommendation is to make sure that the content on the site is 
current and complete. First, include all types of documents (a user manual. Marine Corps 
Order, Marine Corps Bulletin, P and non-P-type orders, standard operating procedures 
and guidebooks, etc.). Second, if the relevant content is available on other sites and it is 
not feasible to include it on the USMC site, I recommend that you provide a “see also” 
feature that indicates other reliable sources for each topic. Finally, all content should be 
dated and source referenced in a manner in which users can ascertain whether additional 
or more current information exists elsewhere. 

Users demonstrated a variety of mental models and key words for locating 
relevant information. To aid them, I recommend an overview page be designed that 
provides a “one-stop shop” for navigating to needed content. This page should be the 
first option on the drop-down menu labeled “Publications.” The page should contain a 
comprehensive list of policy and procedure documents organized according to users’ 
mental models rather than primary office of responsibility or other means. Some of the 
users’ mental models uncovered during this study were to search by topic, search by 
SSIC code, and to view documents by task relevancy. Further research is recommended 
to detail these models more completely. 
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2. 


Improve the www.usmc.mil Search Engine 


Users that used the seareh box on www.usmc.mil found it to be highly ineffective 
and abandoned the site for better search capability elsewhere. Improving the search 
engine would make the site more effective and increase users’ confidence. 

First, users employed common keywords that were driven by the task 
requirements. These common keywords were logical and should have returned 
meaningful results. However, the search results on the USMC site did not do this. For 
example, a search for 'supply manual,' and another for 'accountability,' returned zero 
results even though the Marine Corps Consumer Level Supply Policy Manual (which 
discusses accountability at length) is located on the site. I recommend that the search 
engine be improved or replaced by one that is more robust. 

Additionally, users expect the search results to clearly identify what parts of the 
site were searched and were confused when it did not do this. I recommend that the 
search filter clearly identify what area of the site was searched. Further, the distinction 
between searching directives and orders should be removed unless it is significantly 
relevant to another user group. 

3. Provide Users with Both Site and Document Customization Options 
that Support Their Work 

Users often know which documents they need to consult, and they want those 
documents to be available to them wherever they go. Providing the option for a user to 
create a policy page that is customized to their preferences would enhance the user’s 
experience and better support their work. In this manner, a user who commonly refers to 
only three Marine Corps orders regularly could have a personalized policy homepage 
where those three documents reside. The Federal Acquisition Regulation (FAR) 
homepage even has an option for users to subscribe to receive policy update notices by e- 
mail whenever there is a change. These types of enhancements simplify the users’ tasks 
when accessing policy documents and provide the portability they desire. 
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Users would also benefit from doeument customization options. Specifically, I 
recommend that the site include the ability to highlight text areas, make notes, and 
bookmark pages within a document. These features would provide a useful means for 
users to replicate their paper-based experience and build confidence in using the 
electronic documents. 

Anchoring and grouping is another customization tool that could provide the 
same, if not better, functionality as the ability to tab pages. Anchoring allows users to 
mark locations in text for future retrieval and some tools allow for inline annotations as 
well. Grouping merely refers to the ability of a user to ‘name’ a group of anchors that are 
related in ways that are meaningful to the individual user. For example, if users 
frequently refer to a document that discusses UURI accountability, and that discussion is 
spread out in different chapters or even different documents, as is the case, a user could 
use an anchor tool to mark those locations and tie them together by grouping them under 
a title ‘UURI accounting’. This tool would allow the user to collate information digitally 
so that the answers to frequently asked questions could be referred to easily. 

As a more long-term goal, consideration may be given to a shared knowledge 
system where, collectively, best practices and interaction within the user-community 
could enhance the support system overall. For example, users in this study were asked to 
search for information on how to properly account for UURI SL-3 items. This is not the 
first time this debated subject has plagued a supply officer and required him to refer to 
policy documents to find the answer. A shared knowledge system would not only give 
the user access to the policy documents, but may alleviate the need for him to conduct 
exhaustive research for best practices already established elsewhere. 

4. Provide a Collapsible Structure for Navigating and Hyperlinking 
Internal References 

Users primarily navigated within documents by using a document’s table of 
contents (TOC) as a guide to get them to a section that they thought would be useful. 
However, the current document design proved to be inefficient and required a lot of 
scrolling and returning to the TOC or subheading area to search for alternatives when the 
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user did not find the information they needed. Additionally, when referenees to another 
part of the doeument were eneountered in the reading, users had no way to easily move to 
that area. 

Providing an internal menu strueture, with eollapsible headings and brief eontent 
deseriptions, would allow users to sean topie headings and investigate details easily while 
still maintaining an overall view of the entire doeument strueture. Also, providing 
hyperlinks would allow users to investigate other related material easily. This is 
espeeially useful sinee information is typieally parsed. 

5. Enhance Documents with Contextual Aids 

Users often eited that they needed to ‘eheek’ or ‘double eheek’ to find the loeation 
of the doeument in whieh they were eurrently loeated. Headings and subheadings are a 
good way to indieate this eontext, and this eontextual problem ean be partially averted by 
employing a eollapsible strueture as diseussed. However, eontextual eues ean be made in 
other ways that are benefieial to the user. One example would be to add a navigation 
frame adjaeent to the doeument whieh shows the heading and subheading strueture and 
highlights the eurrent loeation. 

Another eontextual enhaneement option might be to provide ‘preview’ 
information whieh would show the first few lines of the paragraph that a user was 
eontemplating when merely plaeing the mouse pointer over the heading. This would help 
diminish unneeessary searehing and would keep the antieipatory eontext in perspeetive. 

Whatever eontextual aid is employed, it should allow the user to have an 
understanding of the doeument strueture, allow them to see their eurrent loeation within 
that strueture, and set their expeetations on what eontent ean be expeeted when they go to 
a speeifie seetion. 

6. Improve Internal Document Search Capability by Moving Away 
From Current Document Format 

Currently, most eleetronie poliey doeuments are seanned versions of paper poliey 
doeuments. Many, but not all, have undergone some objeet eharaeter reeognition to 
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enable searehing. However, Adobe pdf files have a finite seareh eapability that is limited 
by the Adobe software. Improving the seareh eapability, then, would require the poliey 
doeuments to be migrated to a souree format that lends itself to being searehed more 
effeetively (for example, html or xml). Additionally, seareh engine eompanies like 
Google have produets available to enhanee doeument seareh eapabilities. Any enhaneed 
seareh eapability eonsidered should take into aeeount the users mental models and need 
for eontext awareness as indieated above. 

7. Consider Implementing a More Robust Database 

Users, when looking for information, used the USMC site, Google, and other 
loeations that seemed likely to have the answers. Not only do users want aeeess to all the 
information neeessary to answer their questions, the Marine Corps also wants their 
supply offieers to have information that is aeeurate. A eonsolidated DoD database of 
polieies and proeedures may be the answer to both of these needs. This ehange may 
seem diffieult at the surfaee due to the faet that many different ageneies are responsible 
for different portions of the overall publieation system. However, many industries have 
ineorporated database formats that allow multiple sourees to be effeetively integrated into 
a single user interfaee for presentation to the end user. This is not uneommon and eould 
be aeeomplished for poliey doeuments as well. Additionally, this redesign effort should 
address users’ goals, tasks, and mental models eonsistent with the findings of the eurrent 
study. 


8. Conduct Further Research to Help Identify Additional Problem 
Areas, Underlying Users’ Needs, and Best Practices 

This study was an exploratory study that revealed a number of usability issues. 
Additional user testing ean expand our understanding of the issues and help prioritize 
their impaet. As ehanges are made to the system within an overall redesign effort, I 
reeommend that on-going testing be eondueted. If a major redesign is planned, I 
reeommend that on-going testing be eombined with preliminary user needs analysis and 
iterative user input to help guide the design proeess. 
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In addition, I recommend that the Marine Corps investigate best practices of 
others to help guide site design. One resource for best practices is found on the website 
http://www.usabihty.gov, which provides comprehensive guidelines on the usability of 
web sites. In addition, I recommend looking at other web sites that are intensively policy 
driven. For instance, some of the issues addressed in this report can be seen in a current 
USMC site: USMC Contract Management Process Guide (CMPG) located at 
http://hqinet001.hqmc.usmc.mil/i&l/v2/CMPG/index.htm . This web site supports 
management of the contracting process, including the GCPC, and implements some 
features that may be useful. Specifically, the CMPG site allows users to access policy 
guidance in html format augmented by links to other contributing source information. A 
comprehensive flowchart of the process can be accessed to guide users to the correct area 
where processes are discussed step-by-step with appropriate references available. The 
context tools and improved search capability are also present on this site, which does not 
rely on Adobe files. Additionally, the site provides common templates and other 
resources useful to the user. The site is not customizable, but definitely moves toward 
the knowledge-sharing type system discussed in the current study. 

B. CONCLUSION 

This was an exploratory study, focused on high level usability aspects of a 
complex system. In this study, I chose a specific user group to perform specific tasks 
using a ‘discount’ approach to usability. I have shown how a limited number of 
participants can reveal many significant usability problems that can lead to improved 
design of a product or system. This does not mean that the system is currently unusable 
or that implementation of these recommendations would correct all usability concerns. 
The results of the testing should be considered within the purpose of usability testing 
itself- to make a product or system more usable. To that end, the study has made several 
recommendations for the system under evaluation. 

Recommendations with a low cost-high benefit ratio may be considered relatively 
risk-free to implement and may not require any further research prior to implementation. 
Significant or costly recommendations may benefit from further testing and problem 
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triangulation, where data eolleetion is condueted under multiple methods, measures, or 
approaehes to determine possible convergenee in pereeived problem areas. 
Triangulation can be accomplished through various means including employing different 
research methods (questionnaires, focus groups), multiple evaluators, or multiple user 
groups within a single method. 

Iterative testing is recommended either as a method of triangulation, to conduct a 
more thorough examination of a particular usability area of interest identified within the 
scope of this project, or as a next step after addressing the major findings of this 
exploratory research. 


Chauncey E. Wilson, “Triangulation: the explicit use of multiple methods, measures, and 
approaches for determining core issues in product development,” interactions 13,6 (November 2006), 46- 
ff. DOI= http://doi.acm. 0 rg/lO.l 145/1167948.1167980 

57 Ibid. 
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APPENDIX A. SUS TEST 


Post Test Questionnaire 
SUS 

Mean Seore Response indieated in Bold 


Strongly Strongly 

Disagree Agree 


1. I think that I would like to use this system 
frequently 




4.2 



1 

2 

3 

4 

5 

2. I found the system unneeessarily eomplex 


1.6 





1 

2 

3 

4 

5 

3. I thought the system was easy to use 



3.2 




1 

2 

3 

4 

5 

4. I think that I would need the support of a 
technical person to be able to use this system 

1.2 






1 

2 

3 

4 

5 

5. I found the various functions in this system 
were well integrated 



3.2 




1 

2 

3 

4 

5 

6. I thought there was too much inconsistency 
in this system 



3 




1 

2 

3 

4 

5 

7. I would imagine that most people would 
learn to use this system very quickly 




4 



1 

2 

3 

4 

5 

8. I found the system very cumbersome to use 


1.8 





1 

2 

3 

4 

5 

9. I felt very confident using the system 



3.4 




1 

2 

3 

4 

5 

10. I needed to learn a lot of things before I 
could get going with this system 


1.8 
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APPENDIX B. SURVEY 


SUPPLY AREA IMPORTANCE 


1. Do you consent to this survey? 

Response 

Response 


Percent 

Count 

I consent I 

I 99.0% 

98 

I DO NOT consent Q 

1.0% 

1 


answered question 

99 


skipped question 

10 


2. In the conduct of duties as a supply officer, how much of your time do you spend looking for information in electronic 
publications? 



Response 

Response 


Percent 

Count 

<10% 1 1 

22 7% 

22 

10%-20% 1 1 

41.2% 

40 

20%-30% 1 1 

20 6% 

20 

30%-50% 1 1 

124% 

12 

>50% □ 

31% 

3 


answered question 

97 


skipped question 

12 
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3. What are the three most common reasons you look for information in electronic reference publications? (Select only three) 


Response 

Response 


Percent 

Count 

Develop trainlna for Subordinates 1 1 

12.8% 

13 

Develop Internal procedures 1 1 

23.5% 

24 


complies with reference(s) 

I 53.9% 

55 

Advise Commander 1 

I 59.8% 

61 

Advise 0 €er(s) / commodity areas 1 1 

245% 

25 


---- 

21 6% 

22 

action(s) 


1 578% 

59 

interested party 

Understand a reouired task 1 1 

22.6% 

23 

Plannina 1 1 

17.7% 

18 

Other (please specify) 

0.0% 

0 


answered question 

102 


skipped question 

7 


4. When looking for information in electronic publications, in which subject area do you feel you have spent the most time? 



Response 

Response 



Percent 

Count 

Property Control 

1 1 

29 4% 

30 

Requisition Management 


108% 

11 

Financial Management (GCPC, 

1-1 



contracts, budget, fiscal standing) 




Miscellaneous (POR, ammo acctng, 




personal effects, publication mgmt. 

1' ..1 

22 6% 

23 

clothing, etc) 




Combat Service Support (plan 

[~l 

4 9% 

5 

operations support, MPF, etc) 

1_1 



Storage and Distributioo 

D 

2.0% 

2 



answered question 

102 



skipped question 

7 
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5. When looking for Property Control information, which two tasks require you to refer to publications most often? (select only 
two) 



Response 

Response 


Percent 

Count 

Allowances 1 1 

17.2% 

5 

T/O&E Chanqes 1 1 

17 2% 

5 

Redistributions d 

3.5% 

1 

Equip Disposal O 

3.5% 

1 

Controlled Items 1 1 

10.3% 

3 

Rollbacks 

0.0% 

0 

CMRs 1 1 

44.8% 

13 

Inventory 1 1 

10.3% 

3 

Money Value Gain/Loss 1 1 

13.8% 

4 

Subsidiary records (non-issued 

0.0% 

0 

serialized gear) 



Crane Reoort 1 1 

10.3% 

3 

Temporary loan files 

0.0% 

0 

non-T/E assets 1 1 

6.9% 

2 

Control of Individual Eauioment 1 1 

138% 

4 

Investiqations 1 1 

27 6% 

8 

Assist in maintaining MCGERR i— i 

3.5% 

1 

items ^ 



Analyze orooertv accountina reoorts 1 1 

17.2% 

5 

Other (please specify) 1 1 

6.9% 

2 


answered question 

29 


skipped question 

80 


81 






















6. When looking for Requisitioning information, which two tasks require you to refer to publications most often? (select only 

two) 

Response 

Response 


Percent 

Count 

Processing additional demands |-. 

(requisitions) 

273% 

3 

Uniform Movement Issue and 

Priority System (UMMIPS) 

0.0% 

0 

Maintenance of Requisitions - |- 

45.5% 

5 

follow up 

Manage Daily Maintenance t - 1 

18.2% 

2 

Transactions List - 

Reconcile MOV/backorder validation 1 1 

18.2% 

2 

Manage supply discrepancy report 1 1 

27 3% 

3 

Analyze requisition management |-. 

364% 

4 

lepoils 

Other (please specify) 1 1 

9.1% 

1 


answered question 

11 


skipped question 

98 
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7. When looking for Financial Management information, which two tasks require you to refer to publications most often? (select 
only two) 




Response 

Response 



Percent 

Count 

Prepare budget 

1 1 

16 1% 

5 

Execute budget/spending plan 

1 1 

22 6% 

7 

midyear review 

□ 

3.2% 

1 

Manage fiscal accountability 

CZl 

9.7% 

3 

Annual closeout 

□ 

6.5% 

2 

Manage fiscal for unit funded 

□ 

3.2% 

1 

exercises 




Manage fiscal for externally funded 

□ 

3.2% 

1 

exercises 




Manage reimburseable funding 

□ 

3.2% 

1 

Government credit card program 

1 

1 64.5% 

20 

(GCPC) 




Direct purchases using the 

1-1 



government credit card 

1_I 



Initiate purchasing and contracting 

1 1 

41 9% 

13 

requests 




Manage procurement of supplies 




from DSSC 




Validate collection and turn in of 




government funds 




Other (please specify) 

□ 

3.2% 

1 



answered question 

31 



skipped question 

78 
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8. When looking for Miscellaneous information, which two tasks require you to refer to publications most often? (select only 

two) 






Response 

Response 



Percent 

Count 

Supply officer and CO certificate of 

1-1 

21 7% 

5 

relief 




Endorse outgoing supply officer 

1-1 

8 7% 

2 

certificate of relief 




Letters of appointment and 

1 1 

87% 

2 

revocation 




Maintain results of audits and 

I-1 



verification 

1_1 

liJ.UTb 


MLSR reports 

1 1 

304% 

7 

Ammunition accountability 

□ 

44% 

1 

PORs 

□□ 

8,7% 

2 

Personal effects 

1 

1 65.2% 

15 

Manage permanent issue of special 


0 0% 

0 

equipment 




Individual clothing records 

1 1 

21 7% 

5 

Publications management 


0.0% 

0 

MOS sustainment training 

□ 

44% 

1 

Brief CO on status of supply 

1-1 

174% 

4 

operation 




Other (please specify) 


0.0% 

0 



answered question 

23 



skipped question 

86 


84 

















9. When looking for CSS information, which two tasks require you to refer to publications most often? (select only two) 


Response 

Response 


Percent 

Count 


support MAGTF operations ^ 

I 60.0% 

3 

Provide supply focused combat 

service support input in MPF | | 

20.0% 

1 

operations planning 

Conduct supply support 

operations within a combat service I 

I 60.0% 

3 

support element 

Preparation of supplies for 

deployment 



Accountability of unit MPF assets I I 

20 0% 

1 

Establish a field warehouse 

0.0% 

0 

Other (please specify) I I 

20.0% 

1 


answered quesxion 

5 


skipped question 

104 


10. When looking for Storage and Distribution information, which two tasks require you to refer to publications most often? 
(select only two) 

Response Response 


Maintenance of stored assets 1 

Warehouse safety orocedures 1 

Percent 

1 50.0% 

1 50 0% 

Count 

1 

1 




1 100 0% 

2 

Electrostatic discharge sensitiye i- 

devices - 

1 50.0% 

1 

Radiological control program 

0.0% 

0 

Other (please specify) 

0.0% 

0 


answered question 

2 


skipped question 

107 


85 























11. Thank you for your contribution. Camp Pendieton I San Diego area personnel only - we would appreciate your participation 
in a usabiiity study (which takes no more than one hour). This study is being conducted to better understand issues in 
accessing electronic documents and supply policies, and user participation is critical to gaining that understanding. If you 
would like to participate, piease enter your e-maii address in the box below so we may provide you vrith additional information. 
Volunteers selected will be asked to complete a task requiring publications use, followed by a short interview. 

Response 

Count 

0 

answered question 0 

skipped question 109 
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APPENDIX C. 


PRE-TEST QUESTIONNAIRE 


What is your age group? 

18-25 

26-35 

36-45 

46+ 

What is your current billet? 


Describe your familiarity with the internet? 

Very Familiar -1 use it to look for information, make purchases, download applications 
Familiar -1 use it to look for information, make purchases 
Somewhat Familiar -1 use it primarily to look for information 

In the conduct of your duties as a supply officer, how much of your time do you spend 
looking for information in electronic publications? 

0 % - 10 % 

10 %- 20 % 

20% - 30% 

30% - 50% 

More than 50% 

How long have you been a supply officer? 

0-4 years 
4-10 years 
10 - 15 years 
15 + years 

What is your level of education? 

High school only 
Some College 
BA/S degree 

Some post-graduate education 
Graduate degree or higher 
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APPENDIX D. TEST PLAN 


Scope 


The test will foeus on Marine Corps supply Captains using eleetronie poliey and 
regulatory doeuments related to finaneial management and property eontrol funetions. 

Purpose 

The purpose of this test is to answer the researeh questions: 

o Can users easily and effieiently loeate finaneial management and 
property eontrol regulations? 

o Can users easily and efficiently locate all necessary information to 
answer work-related questions? 

o Can users determine if they have the most current information? 
o How do users look for information? 


Schedule 


The test will be conducted on Monday, 5 November and Tuesday, 6 November 2007, in 
accordance with the following schedule: 

Participant 1: _ 

Participant 2: _ 

Participants: _ 

Participant 4: _ 

Participants: _ 

Participant 6: _ 
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Participants 

Participants will represent the user population, validated by the pre-test questionnaire. 

o Education 

Predominantly hold a BA degree, some higher, but none lower. 

o Job Experience 

Between two and ten years of aetive duty supply offieer experienee 

o Computer Experience 

Familiarity with internet use 

o Demographic 

Predominantly between 26-35 yrs old 

Scenarios 

o Scenario 1 

Lt Umptyfratz, the eommO went to the CO about a disagreement 
with you on his CMR, whieh eontains the following three items: 

TAMCN H2067, serial # 4536 
TAMCN H2067, serial # 4537 
TAMCN A0267, serial # M80300001 

The Lt elaims that the H2067 items are listed as SL-3 items in the 
TM for TAMCN A0267. Therefore, he says, they should not be 
listed separately on his CMR beeause he’s already aeeounting for 
them in the SL-3 inventory. He argues that it is the same situation 
as a MRC-138, where the vehiele is SL-3 to the radio system and 
the vehiele is not aeeounted on the CMR separately. 

You discover through initial researeh that the items are, in fact, 
SL-3. They are listed as UURI, and the quantity says AR. 

The CO ealled down to your offiee just now: “Captain, I’m headed 
to luneh in five minutes, and I’d like to look at the regulations 
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regarding this matter while I’m out. We’ll discuss it further when I 
get back.” 


• Task 1 

Electronically locate all applicable policy/regulation 
document(s) that will help properly guide you in 
accounting for the SL-3 items 

• Task 2 

“Hey Captain, my lunch partner canceled on me. Meet me 
up here in ten minutes to discuss the CMR situation.” 

Using the electronic policy document(s), identify how to 
properly account for this SL-3 item. 

o Scenario 2 

You are the supply officer at an infantry battalion located at Camp 
Pendleton. Your battalion has been mobilized and is conducting 
work-up training in 29 Palms, CA, so no maintenance personnel 
are available. A GSA vehicle is broken and requires service. The 
CO doesn’t want to wait for you to send it somewhere on base for 
repair, so he suggests taking it to the commercial GM dealer down 
the road and using the GCPC card to pay for the service instead. 
The GM dealer estimates the cost at $2600, but couldn’t guarantee 
it wouldn’t be more than that. The CO is pretty strict and always 
wants to see the exact verbiage and the reference source when you 
provide information to him. Using only electronic access to policy 
documents as your source, what would you advise the CO? 

Key data: 

Estimated Cost: $2600, no ceiling price established 
Purchase Type: unpriced Service 
Item: GSA vehicle repair 

• Task 3 


The CO just called and said he’ll be down in five minutes. 
He asked you to electronically locate all applicable 
policy/regulation document(s) that you think will apply to 
this situation. 
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• Task 4 


The CO tells you he spoke to a friend who told him that the 
key eoneems with this purehase would be to: 

Determine whether the purehase exeeeds the GCPC 
purchase amount threshold? 

Determine if the GCPC regulations/policy prohibit or limit 
a purchase of this type? 

Determine if any authority prohibits or limits the purchase 
for this particular item? 

• Completion 

This completes scenario 2. 


Data collected 


o Questionnaires 

• PRE-TEST QUESTIONNAIRE [APPENDIX Cl 

The purpose of the questionnaire is to record information 
about the participants of this study as they compare to 
characteristics of the entire user population to ensure a 
representative group is tested. 

• Post Test questionnaire (SUS) [APPENDIX A] 

The purpose of the SUS is to solicit the user’s general 
assessment of overall usability. 

• Videotape 

Video tape all sessions 

• Observation Notes 


Make observation notes during testing to refer to during 
interview 
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Interview Questions 


Not ‘was it easy’ - ean you talk about your experience 
with....? 

Tell me how you know.how you. 

Can you talk about that more? 

How confident are you....? 

Execution 


o Pre-test 

Setup 

o Test 


• Introduction 


Welcome, and thank you for participating today. My name 
is Captain Scott Stahl. I am a student at the Naval 
Postgraduate School, and I’m working on a project to 
evaluate how well electronic publications assist you in 
doing your work. As part of this project, I am observing a 
variety of supply officers conduct various tasks using 
electronic publications to see what elements of the design 
might need to be improved. 

I’d like to stress that I’m testing the publication system, and 
not your abilities. Also, I do not represent your command, 
or any other command, for that matter. So don’t worry 
about making mistakes. There is no right or wrong answer. 
If you find parts of the system difficult to use or 
understand, it is likely that other people do also, and my 
purpose here is to identify these items and make 
appropriate recommendations for changes to improve the 
system. 

If you ever feel that you are lost or cannot complete a 
scenario with the information that you have been given, 
please let me know. I’ll ask you what you might do in a 
real-world setting and then either put you on the right track 
or move you on to the next scenario. 
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Finally, as you use the system, please do so as you would at 
work. I do ask that when looking for information, you do 
so as quiekly and as accurately as you can. 

Overview 


During this test, I will first ask you to fill out consent forms 
and a pre-test questionnaire. 

Next, you will be presented with 2 scenarios containing 4 
tasks each. The scenarios will require you to navigate 
electronic publications. Each scenario will be presented 
individually and last 10 minutes. As you work through 
them, I will observe and ask you to ‘think aloud’ as you 
work. 

I will also videotape the session, recording only the 
computer screen for analysis in depth at a later time. 

Upon completion, a final interview and questionnaire will 
be administered. 

In total, the session should only last about one hour. If you 
would like to take a break at any time, just let me know. 

Do you have any questions at this time? 

Consent and Pre-Test Forms 


At this time, I would like to ask for your consent to 
participate and be videotaped. 

Consent Form 
Videotaping Consent 

Next, this pre-test questionnaire asks about characteristics 
that will be compared with the user population to validate 
that participants actually represent the users. 

Pre-Test Questionnaire 

Discuss Think Aloud 


[think aloud example here]. 








• Initiate Scenario 1 


At this time, we are ready to begin. You will be given 10 
minutes to complete the scenario. During the scenario, if 
you reach a point where you would normally stop and 
phone a colleague for assistance or quit altogether, please 
state that and we will move on to the next item. 

[ensure videotape] 

[Present scenario 1] 

[keep time] 

[notes] 

[end time] 

Thank you. That completes scenario one. 

• Initiate Scenario 2 


We will now conduct scenario 2. You will be given 10 
minutes to complete the scenario. During the scenario, if 
you reach a point where you would normally stop and 
phone a colleague for assistance or quit altogether, please 
state that and we will move on to the next item. 

[ensure videotape] 

[present scenario 2] 

[keep time] 

[notes] 

[end time] 

Thank you. That completes scenario two. 


Post-test 


o Questionnaire 


I have a brief questionnaire here that I’d like you to complete also. 
The information you provide is for our use only. Your name is not 
stored with the questionnaire data. 

o SUS TEST 
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o Interview 


Now that we have completed the scenarios, I have a few questions 
about what you experienced during the test. 

Interview Questions 


Debrief 


This concludes our session. Once again, I’d like to say tha nk s for 
coming today. 
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